Horizontal Pod Autoscaling

With Horizontal Pod Autoscaling, Kubernetes automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization (or, with alpha support, on some other, application-provided metrics).

The current stable version, which only includes support for CPU autoscaling, can be found in the autoscaling/v1 API version. The alpha version, which includes support for scaling on memory and custom metrics, can be found in autoscaling/v2alpha1 in 1.7 and autoscaling/v2beta1 1.8 and 1.9.

Kops can assist in setting up HPA and recommends Kubernetes 1.7.x to 1.9.x and Kops >=1.7. Relevant reading you will need to go through:

While the above links go into details on how Kubernetes needs to be configured to work with HPA, a lot of that work is already done for you by Kops. Specifically:

  • Enable the Aggregation Layer via the following kube-apiserver flags:
  • --requestheader-client-ca-file=<path to aggregator CA cert>
  • --requestheader-allowed-names=aggregator
  • --requestheader-extra-headers-prefix=X-Remote-Extra-
  • --requestheader-group-headers=X-Remote-Group
  • --requestheader-username-headers=X-Remote-User
  • --proxy-client-cert-file=<path to aggregator proxy cert>
  • --proxy-client-key-file=<path to aggregator proxy key>
  • Enable Horizontal Pod Scaling ... set the appropriate flags for kube-controller-manager:
  • --horizontal-pod-autoscaler-use-rest-clients should be true.
  • --kubeconfig <path-to-kubeconfig>

Ensure that you use 1.8.5 at minimum for Kubernetes 1.8.x due to a nasty bug affecting the API aggregation layer. It was fixed in kubernetes/kubernetes#55259.

Cluster Configuration

Support For Multiple Metrics

Enable API versions required to support scaling on cpu, memory and custom metrics:

# On K8s 1.7
spec:
  kubeAPIServer:
    runtimeConfig:
      autoscaling/v2alpha1: "true"

# On K8s 1.8 and 1.9
spec:
  kubeAPIServer:
    runtimeConfig:
      autoscaling/v2beta1: "true"

If you've set the above configuration, your cluster is now ready for the resource metrics API (installation instruction here). The compatibility matrix is as follows:

Metrics Server Metrics API group/version Supported Kubernetes version
0.3.x metrics.k8s.io/v1beta1 1.8+
0.2.x metrics.k8s.io/v1beta1 1.8+
0.1.x metrics/v1alpha1 1.7

Support For Custom Metrics

Enable gathering custom metrics:

spec:
  kubelet:
    enableCustomMetrics: true

Note that the deprecated and inactive option '--enable-custom-metrics' has been removed in Kubernetes 1.11 #60699

Enable the horizontal pod autoscaler REST client:

spec:
  kubeControllerManager:
    horizontalPodAutoscalerUseRestClients: true

If you've set the above configuration, your cluster is now ready for the custom metrics API. Register it via the API aggregation layer. If you're using Prometheus, checkout the custom metrics adapter for Prometheus.

Implementation Details

These are the PRs that enable the required configuration:

  • kubernetes/kops#3679 - sets --requestheader-xxx kube-apiserver flags required to enable aggregation layer 
    --requestheader-client-ca-file=<path to aggregator CA cert>
--requestheader-allowed-names=aggregator
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-group-headers=X-Remote-Group
--requestheader-username-headers=X-Remote-User
  • kubernetes/kops#3165 - sets --proxy-client-xxx kube-apiserver flags required to enable aggregation layer 
    --proxy-client-cert-file=<path to aggregator proxy cert>
--proxy-client-key-file=<path to aggregator proxy key>
  • kubernetes/kops#3939 - add config option to set --horizontal-pod-autoscaler-use-rest-clients kube-controller-manager flag required to enable custom metrics
  • kubernetes/kops#1574 - add config options to set --enable-custom-metrics flag on master and node kubelets required to enable custom metrics