Kubernetes Horizontal Pod Autoscaling with external metrics API
Table of Contents
Introduction
This is the third article in a series dedicated to installing and configuring Kubernetes Horizontal Pod Autoscaling:
- Horizontal Pod Autoscaling with metrics API
- Horizontal Pod Autoscaling with custom metrics API
- Horizontal Pod Autoscaling with external metrics API
In the first two articles I gave an overview of Kubernetes HPA and showed how to configure autoscaling with metrics API and custom metrics API server running locally with minikube.
In this article, we will continue exploring Horizontal Pod Autoscaling and this time we will use external metrics API.
Source code for the demo app and kubernetes objects can be found here.
Overview of HPA with external metrics API
External metric type for autoscaling is based on metrics coming from outside of Kubernetes cluster (message queue, 3rd party application, etc). Please read more at the official documentation page.
┌───────────┐ ┌──────────────────────┐
│ │ external/metics.k8s.io │ │
│ HPA ├────────────────────────────────► External Metrics API │
│ │ │ │
└─────┬─────┘ └──────────┬───────────┘
│ │
│ ┌──────────▼───────────┐
┌─────▼─────┐ │ │
│ │ │ Prometheus Adpater │
│ Deployment│ │ │
│ │ └──────────┬───────────┘
└─────┬─────┘ │
│ ┌──────────▼───────────┐
│ │ │
┌─────▼─────┐ │ Prometheus │
│ │ │ │
│ ReplicaSet│ └──────────┬───────────┘
│ │ │
┌──┴──────┬────┴─────────┐ │ /metrics
│ │ │ │
┌─────▼──┐ ┌─▼──────┐ ┌───▼────┐ ┌───────────▼────────────┐
│ │ │ │ │ │ │ Message Queue │
│ POD1 │ │ POD2 │ │ PODN │ │ 3rd party application │
│ │ │ │ │ │ │ etc │
└────────┘ └────────┘ └────────┘ └────────────────────────┘
Configuring HPA with external metrics API
Requirements
- Kubernetes cluster with installed:
- Prometheus operator
- Prometheus adapter
- Kubectl
- Helm
Starting Kubernetes cluster, installing kubectl and helm
- To start kubernetes cluster and install kubectl please refer to the first article.
- Helm is a package manager for Kubernetes, it allows to manage complex application deployments with templates. Please follow the official instructions to install
helmcli. Check if you have successfully installed helm:
$ helm version
version.BuildInfo{Version:"v3.7.1", GitCommit:"1d11fcb5d3f3bf00dbe6fe31b8412839a96b3dc4", GitTreeState:"clean", GoVersion:"go1.17.2"}
Installing Prometheus server
Instructions on prometheus server installation were given in the last article, please refer this page.
Installing Prometheus Adapter and enabling external metrics API
Prometheus Adapter implements Kubernetes Custom, Resource and External Metrics APIs, for more details please check its source code repo. We are installing prometheus adapter in the same names with other prometheus components, please note that we need to override default prometheus url via helm install --set flag:
$ helm -n prometheus-resources upgrade --install prometheus-adapter prometheus-community/prometheus-adapter \
--version=4.1.1 \
--set prometheus.url=http://kube-prometheus-stack-prometheus \
--values=https://raw.githubusercontent.com/ilyamochalov/source-code-mics/main/k8s/HPA/prometheus-adapter-enable-external-metrics.yaml
If helm install printed success message, then we can check for available exteraenl metrics using the following command:
$ kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "external.metrics.k8s.io/v1beta1",
"resources": []
}
Adding extra scrape configurations to prometheus
Here we need to make sure our prometheus has external metrics. You can use two way:
- configure our cluster Prometheus to scrape external data
- configure external Prometheus server that has you data to do remote write to our cluster Prometheus
Configuring HPA with external metrics for your app
To expose this metics to the metrics server we need to update Prometheus adapter rules config map.
externalRules:
- seriesQuery: 'rabbitmq_queue_messages_ready{}' # example
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (vhost)
resources:
overrides: { namespace: {resource: "namespace"} }
name:
as: rabbitmq_queue_messages_ready
Let’s apply required rules changes:
$ kubectl apply -f https://raw.githubusercontent.com/ilyamochalov/source-code-mics/main/k8s/HPA/external-metrics-demo-app/adapter-rules.yaml
$ kubectl -n prometheus-resources rollout restart deploy/prometheus-adapter
For the full description of Prometheus adapter rules and how to work with it please read its Configuration Walkthroughs and Config documentation pages.
Configuring HPA resource with external metrics
Add HPA Configuration. The HPA rule states that we want to scale up our deployment when average number task being processed by all pods is higher then 10. Create it with kubectl apply -f https://raw.githubusercontent.com/ilyamochalov/source-code-mics/main/k8s/HPA/external-metrics-demo-app/k8s-hpa.yaml:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: external-metrics-task-demo
namespace: hpa-demo
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: external-metrics-task-demo
minReplicas: 1
maxReplicas: 5
metrics:
- type: External
external:
metric:
name: rabbitmq_queue_messages_ready
selector:
matchLabels:
vhost: "your-host-name"
queue: "my-queue-name"
target:
type: AverageValue
averageValue: 40
HPA configurations
Please read HPA kubernetes API to see full list of available options.
Summary
Horizontal Pod Autoscaling with external metrics API helps to scale based on metrics scraped from external sources.
