Kubernetes Horizontal Pod Autoscaling with external metrics API

Table of Contents


This is the third article in a series dedicated to installing and configuring Kubernetes Horizontal Pod Autoscaling:

  1. Horizontal Pod Autoscaling with metrics API
  2. Horizontal Pod Autoscaling with custom metrics API
  3. Horizontal Pod Autoscaling with external metrics API

In the first two articles I gave an overview of Kubernetes HPA and showed how to configure autoscaling with metrics API and custom metrics API server running locally with minikube.

In this article, we will continue exploring Horizontal Pod Autoscaling and this time we will use external metrics API.

Source code for the demo app and kubernetes objects can be found here.

Overview of HPA with external metrics API

External metric type for autoscaling is based on metrics coming from outside of Kubernetes cluster (message queue, 3rd party application, etc). Please read more at the official documentation page.

         ┌───────────┐                                ┌──────────────────────┐
         │           │      external/metics.k8s.io    │                      │
         │    HPA    ├────────────────────────────────► External Metrics API │
         │           │                                │                      │
         └─────┬─────┘                                └──────────┬───────────┘
               │                                                 │
               │                                      ┌──────────▼───────────┐
         ┌─────▼─────┐                                │                      │
         │           │                                │  Prometheus Adpater  │
         │ Deployment│                                │                      │
         │           │                                └──────────┬───────────┘
         └─────┬─────┘                                           │
               │                                      ┌──────────▼───────────┐
               │                                      │                      │
         ┌─────▼─────┐                                │      Prometheus      │
         │           │                                │                      │
         │ ReplicaSet│                                └──────────┬───────────┘
         │           │                                           │
      ┌──┴──────┬────┴─────────┐                                 │ /metrics
      │         │              │                                 │
┌─────▼──┐    ┌─▼──────┐   ┌───▼────┐                ┌───────────▼────────────┐
│        │    │        │   │        │                │   Message Queue        │
│  POD1  │    │  POD2  │   │  PODN  │                │  3rd party application │
│        │    │        │   │        │                │        etc             │
└────────┘    └────────┘   └────────┘                └────────────────────────┘

Configuring HPA with external metrics API


  • Kubernetes cluster with installed:
    • Prometheus operator
    • Prometheus adapter
  • Kubectl
  • Helm

Starting Kubernetes cluster, installing kubectl and helm

  1. To start kubernetes cluster and install kubectl please refer to the first article.
  2. Helm is a package manager for Kubernetes, it allows to manage complex application deployments with templates. Please follow the official instructions to install helm cli. Check if you have successfully installed helm:
$ helm version
version.BuildInfo{Version:"v3.7.1", GitCommit:"1d11fcb5d3f3bf00dbe6fe31b8412839a96b3dc4", GitTreeState:"clean", GoVersion:"go1.17.2"}

Installing Prometheus server

Instructions on prometheus server installation were given in the last article, please refer this page.

Installing Prometheus Adapter and enabling external metrics API

Prometheus Adapter implements Kubernetes Custom, Resource and External Metrics APIs, for more details please check its source code repo. We are installing prometheus adapter in the same names with other prometheus components, please note that we need to override default prometheus url via helm install --set flag:

$ helm -n prometheus-resources upgrade --install prometheus-adapter prometheus-community/prometheus-adapter \
 --version=4.1.1 \
 --set prometheus.url=http://kube-prometheus-stack-prometheus \

If helm install printed success message, then we can check for available exteraenl metrics using the following command:

$ kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": []

Adding extra scrape configurations to prometheus

Here we need to make sure our prometheus has external metrics. You can use two way:

Configuring HPA with external metrics for your app

To expose this metics to the metrics server we need to update Prometheus adapter rules config map.

    - seriesQuery: 'rabbitmq_queue_messages_ready{}' # example
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (vhost)
        overrides: { namespace: {resource: "namespace"} }
        as: rabbitmq_queue_messages_ready

Let’s apply required rules changes:

$ kubectl apply -f https://raw.githubusercontent.com/ilyamochalov/source-code-mics/main/k8s/HPA/external-metrics-demo-app/adapter-rules.yaml
$ kubectl -n prometheus-resources rollout restart deploy/prometheus-adapter

For the full description of Prometheus adapter rules and how to work with it please read its Configuration Walkthroughs and Config documentation pages.

Configuring HPA resource with external metrics

Add HPA Configuration. The HPA rule states that we want to scale up our deployment when average number task being processed by all pods is higher then 10. Create it with kubectl apply -f https://raw.githubusercontent.com/ilyamochalov/source-code-mics/main/k8s/HPA/external-metrics-demo-app/k8s-hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
  name: external-metrics-task-demo
  namespace: hpa-demo
    apiVersion: apps/v1
    kind: Deployment
    name: external-metrics-task-demo
  minReplicas: 1
  maxReplicas: 5
  - type: External
        name: rabbitmq_queue_messages_ready
            vhost: "your-host-name"
            queue: "my-queue-name"
        type: AverageValue
        averageValue: 40

HPA configurations

Please read HPA kubernetes API to see full list of available options.


Horizontal Pod Autoscaling with external metrics API helps to scale based on metrics scraped from external sources.

Ilya Mochalov
Ilya Mochalov

Helping improve software delivery, operations, and security