# Autoscaling Checks

Autoscaling Checks are small canary workloads and runner Pods that validate the autoscaling control plane in a real cluster. They check that signal generation, metric delivery, KEDA, the selected pod autoscaler, and Deployment replica changes work together. They are not intended to test application business logic.

The chart uses native HPA by default. Compatible versions can validate KPA by setting `autoscaler.class=kpa`, with an optional `autoscaler.syncPeriodSeconds` override. See [Install and Configure Multi-tenant Scaling](https://docs.kedify.io/how-to/multitenant-scaling/) for the required KEDA/KPA setup and the [KPA feature guide](https://docs.kedify.io/features/kedify-pod-autoscaler/) for workload-level controls.

The checks expose `kedify_autoscaling_check_duration_seconds`, a gauge with the latest complete check iteration duration in seconds. Use it for alerts when autoscaling takes longer than your operating threshold.

See [Monitoring](https://docs.kedify.io/observability/monitoring/) to collect check results in Datadog, Prometheus or another monitoring stack.

## What Gets Installed

The Helm chart installs separate runner and target apps for each enabled check. HTTP and CPU are enabled by default. Prometheus, OTel, and memory are opt-in because they depend on additional cluster setup or take longer to settle.

| Check | Signal source | Validates |
| --- | --- | --- |
| HTTP | In-cluster HTTP requests | Kedify HTTP scaler metric, pod-autoscaler metric/activity, scale-out/in |
| Prometheus | Work-simulator sample app metric | Source Prometheus metric, KEDA metric, pod-autoscaler metric, scale-out/in |
| OTEL | Synthetic OTLP gauge from the runner | OTel add-on source metric, KEDA metric, pod-autoscaler metric, scale-out/in |
| CPU | Load-generator sample app CPU profile | `metrics.k8s.io`, pod-autoscaler CPU metric/activity, scale-out/in |
| Memory | Load-generator sample app memory load | `metrics.k8s.io`, pod-autoscaler memory metric/activity, scale-out/in |

## Architecture

Each check has its own runner and target. The runner creates the autoscaling signal, validates the metric and selected HPA or KPA path, waits for the target to scale out and back in, and exposes `kedify_autoscaling_check_*` metrics.

[![Each runner generates load, verifies metrics, observes scale-out and scale-in, then reports the complete check result.](https://docs.kedify.io/assets/images/docs/autoscaling-check-sequence.svg)](https://docs.kedify.io/assets/images/docs/autoscaling-check-sequence.svg)

Scroll to exploreDiagram description

Each autoscaling check has a runner and a target workload. The runner generates a signal, verifies source and autoscaler metrics, observes scale-out, removes or reduces load, and verifies scale-in. The runner exposes check results and the latest complete iteration duration. Signal adapters differ for HTTP, Prometheus, OTel, CPU and memory; the shared verification sequence remains the same.

The signal path varies by check: HTTP uses the proxy and HTTP scaler; Prometheus scrapes the work-simulator metric; OTel receives an OTLP gauge. CPU and memory use Kubernetes `metrics.k8s.io`. The runner observes the selected HPA or KPA and the target Deployment for each path. The table above lists the source and evidence for each check.

## Prerequisites

- Kedify/KEDA is installed in the cluster.

- The Kedify HTTP scaler is installed if the HTTP check is enabled.

- `metrics.k8s.io` is available if the CPU or memory check is enabled.

- Prometheus is reachable from the check runner Pods if the Prometheus check is enabled.

- Prometheus scrapes the bundled work-simulator target if the Prometheus check is enabled.

- The Kedify OTel add-on is reachable from the check runner Pods if the OTel check is enabled.

- The Kedify Agent version supports autoscaling-check Service discovery.

## Install

Install the default HTTP and CPU checks from the [OCI chart on GHCR](https://github.com/orgs/kedify/packages/container/package/charts%2Fautoscaling-checks):

```bash
helm upgrade --install autoscaling-checks oci://ghcr.io/kedify/charts/autoscaling-checks \
  --version <version> \
  --namespace keda \
  --create-namespace
```

Enable the memory check when you want to cover memory resource scaling too:

```bash
helm upgrade --install autoscaling-checks oci://ghcr.io/kedify/charts/autoscaling-checks \
  --version <version> \
  --namespace keda \
  --create-namespace \
  --set memory.enabled=true
```

On a compatible Kedify KEDA/KPA installation, run the same checks through KPA:

```bash
helm upgrade --install autoscaling-checks oci://ghcr.io/kedify/charts/autoscaling-checks \
  --version <version> \
  --namespace keda \
  --create-namespace \
  --set autoscaler.class=kpa \
  --set autoscaler.syncPeriodSeconds=15 \
  --set memory.enabled=true
```

`autoscaler.class` defaults to `hpa`. The optional sync period is applied only in KPA mode. Metric labels continue to use `hpa_metric` and `hpa_active` for both autoscalers; use the exported `autoscaler_class` label to distinguish HPA from KPA.

Enable or disable checks explicitly:

```bash
helm upgrade --install autoscaling-checks oci://ghcr.io/kedify/charts/autoscaling-checks \
  --version <version> \
  --namespace keda \
  --create-namespace \
  --set prometheus.enabled=true \
  --set otel.enabled=true \
  --set otel.addonNamespace=keda \
  --set memory.enabled=true \
  --set cpu.enabled=false
```

The `keda` namespace is recommended. Other namespaces work as long as the runner Services keep the chart labels and the Kedify Agent can read Services.

## Configure Prometheus

Set the Prometheus HTTP API address and the source query for the Prometheus check:

```bash
helm upgrade --install autoscaling-checks oci://ghcr.io/kedify/charts/autoscaling-checks \
  --version <version> \
  --namespace keda \
  --create-namespace \
  --set prometheus.enabled=true \
  --set prometheus.serverAddress=http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090 \
  --set 'prometheus.sourceQuery=sum(work_simulator_inprogress_tasks)'
```

If you use Prometheus Operator, enable ServiceMonitors for the runner metrics and the Prometheus target sample app:

```bash
helm upgrade --install autoscaling-checks oci://ghcr.io/kedify/charts/autoscaling-checks \
  --version <version> \
  --namespace keda \
  --create-namespace \
  --set monitoring.serviceMonitor.enabled=true \
  --set prometheus.targetServiceMonitor.enabled=true
```

If your Prometheus uses pod annotations, the chart enables scrape annotations by default with `monitoring.scrapeAnnotations=true`.

## Configure OTEL

The OTel check uses the [`kedify-otel` scaler](https://docs.kedify.io/scalers/otel-scaler/) and the Kedify OTel add-on. The runner pushes a synthetic OTLP gauge to the add-on, queries the add-on REST API for the source metric, and then validates the KEDA external metric and selected pod-autoscaler path.

Enable the OTel check and point it at the namespace where the OTel add-on Service runs:

```bash
helm upgrade --install autoscaling-checks oci://ghcr.io/kedify/charts/autoscaling-checks \
  --version <version> \
  --namespace keda \
  --create-namespace \
  --set otel.enabled=true \
  --set otel.addonNamespace=keda \
  --set otel.addonServiceName=kedify-otel-scaler
```

By default, the check expects the add-on Service to be named `kedify-otel-scaler` and uses these in-cluster endpoints:

```text
kedify-otel-scaler.<otel.addonNamespace>.svc:4317
http://kedify-otel-scaler.<otel.addonNamespace>.svc:9090
kedify-otel-scaler.<otel.addonNamespace>.svc:4318
```

Override the endpoints only if your add-on is exposed differently:

```bash
--set otel.runner.otlpEndpoint=keda-otel-scaler.keda.svc:4317 \
--set otel.runner.restURL=http://keda-otel-scaler.keda.svc:9090 \
--set otel.scaledObject.scalerAddress=keda-otel-scaler.keda.svc:4318
```

## Dashboard Visibility

The Kedify Agent discovers autoscaling-check runner Services automatically. No extra metrics endpoint setting is required.

The runner Services must have these labels:

```text
app.kubernetes.io/name=autoscaling-checks
app.kubernetes.io/part-of=autoscaling-checks
autoscaling-checks.kedify.io/role=runner
```

The Helm chart sets these labels and exposes a Service port named `metrics`. The Dashboard shows the results in the cluster detail view under **Scaling Health**, including the actual autoscaler class when the installed check version reports it.

## Alerting

Use `kedify_autoscaling_check_duration_seconds` for the primary alert. It records the latest complete iteration duration and includes bounded labels for the overall result and each step.

Example Prometheus alert:

```yaml
groups:
  - name: kedify-autoscaling-checks
    rules:
      - alert: KedifyAutoscalingCheckSlow
        expr: kedify_autoscaling_check_duration_seconds > 300
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: Kedify autoscaling check is slow
          description: Check {{ $labels.check }} took {{ $value }}s in the latest iteration.
```

Use the step labels and `kedify_autoscaling_check_step_duration_seconds` to identify whether the delay is in signal generation, source metrics, KEDA metrics, pod-autoscaler metrics or activity, scale-out, or scale-in. The `hpa_metric` and `hpa_active` label values remain stable compatibility identifiers for both HPA and KPA.

## Verify

Wait for the runner and target Deployments:

```bash
kubectl -n keda rollout status deploy/autoscaling-checks-check-http-runner
kubectl -n keda rollout status deploy/autoscaling-checks-check-http-target
kubectl -n keda rollout status deploy/autoscaling-checks-check-cpu-runner
kubectl -n keda rollout status deploy/autoscaling-checks-check-cpu-target
```

If optional checks are enabled, wait for those runners and targets too:

```bash
kubectl -n keda rollout status deploy/autoscaling-checks-check-prometheus-runner
kubectl -n keda rollout status deploy/autoscaling-checks-check-prometheus-target
kubectl -n keda rollout status deploy/autoscaling-checks-check-otel-runner
kubectl -n keda rollout status deploy/autoscaling-checks-check-otel-target
kubectl -n keda rollout status deploy/autoscaling-checks-check-memory-runner
kubectl -n keda rollout status deploy/autoscaling-checks-check-memory-target
```

Inspect one runner directly:

```bash
kubectl -n keda port-forward svc/autoscaling-checks-check-http-runner 8080:8080
curl -s localhost:8080/healthz
curl -s localhost:8080/metrics
```

On a KPA-enabled installation, include the generated KPA in the verification:

```bash
kubectl -n keda get scaledobject,kpa,hpa,deploy,svc -l app.kubernetes.io/part-of=autoscaling-checks
```

On an HPA-only cluster without the KPA CRD, use:

```bash
kubectl -n keda get scaledobject,hpa,deploy,svc -l app.kubernetes.io/part-of=autoscaling-checks
```

## Troubleshooting

If the dashboard tab is empty, verify that the agent can read Services and that the runner Services have the labels shown above and a `metrics` port.

If the Prometheus check fails, query Prometheus directly with the configured `prometheus.sourceQuery` and confirm it returns the expected value during a check run.

If the OTel check fails before the KEDA metric step, confirm that the runner can reach the OTel add-on OTLP endpoint and REST API in `otel.addonNamespace`.

If the CPU or memory check fails before scaling out, confirm that `metrics.k8s.io` returns Pod resource metrics in the `keda` namespace.

If scale-in takes too long, check the ScaledObject cooldown values and the chart `common.scaleDownTimeout` setting.

## Uninstall

```bash
helm uninstall autoscaling-checks --namespace keda
```

## Continue with this topic

**Configure and verify:** [Collect diagnostics for support](https://docs.kedify.io/troubleshooting/collecting-kedify-configuration/).

**Reference:** [Metrics, status and health signals](https://docs.kedify.io/reference/metrics/).

**Related capabilities:** [Workload diagnostics](https://docs.kedify.io/troubleshooting/workload-scaling/) · [Troubleshoot Kedify](https://docs.kedify.io/troubleshooting/).

---
Canonical: https://docs.kedify.io/features/autoscaling-checks/
Source: src/content/docs/features/autoscaling-checks.md
Documentation index: https://docs.kedify.io/llms.txt