Metrics, status and health signals

Metrics are emitted by specific installed components. Check units, labels and sample freshness before comparing control-plane health with application success. Use Workload diagnostics for the measurement workflow.

For endpoints and collection with Datadog, Prometheus and other stacks, see Monitoring.

Source and enablement

Verify the component’s /metrics output for the installed release. Missing telemetry does not by itself mean the scaling control loop failed.

Emitter	Evidence	Enablement / task
Check runners	End-to-end canary timing and step outcomes	Install and verify checks
HTTP proxy/interceptor	Request rate, concurrency, errors and route ownership	HTTP log/metric diagnosis
Agent PRA controller	Resource decisions, blocked changes, deltas and evaluation timing	Enable PRA
HPA/KPA and custom resources	Desired/current state and conditions	Kubernetes API catalog; exact status fields stay with each kind

Member-cluster CA expiry

The Agent exposes kedify_agent_kedifyconfiguration_multicluster_ca_expiration_timestamp_seconds for parseable CA certificates in member kubeconfigs. See multi-cluster monitoring for labels, days-remaining queries and expiry alerts.

Cardinality and interpretation

Keep namespace/workload/container dimensions bounded to the workloads you operate; do not add request IDs, error messages or timestamps as metric labels. Check-runner result labels are a bounded vocabulary. A fresh healthy canary does not establish that a business request, model or queue worker meets its objective. HTTP traces provide a separate request-timing view.

Every runner exposes /healthz and /metrics.

For Kedify Dashboard ingestion, the Kedify agent auto-discovers labeled runner Services and relays these metrics with kedify_agent_id, which the dashboard API uses to scope results to the selected cluster.

Primary Metric

kedify_autoscaling_check_duration_seconds{
  check="http",
  result="success",
  signal="success",
  source_metric="skipped",
  keda_metric="success",
  hpa_metric="success",
  hpa_active="success",
  scale_up="success",
  scale_down="success"
} 42.5

This gauge is the primary alerting metric. It records the duration of the latest complete check iteration, in seconds. Alert on this metric when autoscaling control-plane latency exceeds your threshold, for example:

kedify_autoscaling_check_duration_seconds > 300

Example Prometheus alert rule:

groups:
  - name: kedify-autoscaling-checks
    rules:
      - alert: KedifyAutoscalingCheckSlow
        expr: kedify_autoscaling_check_duration_seconds > 300
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: Kedify autoscaling check is slow
          description: Check {{ $labels.check }} took {{ $value }}s in the latest iteration.

The result and step labels explain what happened during that iteration. Label values are bounded to:

success | failure | skipped | unknown

Dynamic object names, timestamps, reasons, and error messages are not emitted as labels. Result labels are bounded; kedify_autoscaling_check_info is the static metadata exception and exposes bounded values such as the runner namespace, primary metric name, and interval.

Supporting Metrics

kedify_autoscaling_check_step_duration_seconds{check="http",step="keda_metric"}
kedify_autoscaling_check_last_run_timestamp_seconds{check="http"}
kedify_autoscaling_check_next_run_timestamp_seconds{check="http"}
kedify_autoscaling_check_info{check="http",runner_namespace="keda",primary_metric="kedify_autoscaling_check_duration_seconds",interval_seconds="300",autoscaler_class="hpa"}
kedify_autoscaling_check_metric_threshold{check="prometheus",metric="source"}
kedify_autoscaling_check_observed_metric_value{check="prometheus",metric="source"}
kedify_autoscaling_check_observed_metric_value{check="otel",metric="keda"}
kedify_autoscaling_check_observed_replicas{check="cpu",phase="scale_up"}
kedify_autoscaling_check_observed_replicas{check="memory",phase="scale_up"}

Metric types:

• kedify_autoscaling_check_duration_seconds: gauge, latest complete iteration duration.
• kedify_autoscaling_check_step_duration_seconds: gauge, latest step duration.
• kedify_autoscaling_check_last_run_timestamp_seconds: gauge, Unix timestamp of the latest iteration.
• kedify_autoscaling_check_next_run_timestamp_seconds: gauge, Unix timestamp of the next scheduled iteration.
• kedify_autoscaling_check_info: gauge, static runner metadata for the check.
• kedify_autoscaling_check_metric_threshold: gauge, minimum value used to decide whether the metric check passes.
• kedify_autoscaling_check_observed_metric_value: gauge, observed source, KEDA, or selected pod-autoscaler metric value.
• kedify_autoscaling_check_observed_replicas: gauge, observed target replica count.

Metric threshold and observed value labels use static names such as:

• source
• keda
• hpa

The hpa source and the hpa_metric/hpa_active result labels are retained for dashboard and alert compatibility. When autoscaler.class=kpa, they represent the HPA-compatible KPA status read directly from KedifyPodAutoscaler.

kedify_autoscaling_check_info includes autoscaler_class="hpa|kpa" so consumers can distinguish which control-plane path each runner validates without changing the compatibility step labels.

Replica phases use:

• scale_up
• scale_down

Check Semantics

HTTP:

• signal: in-cluster HTTP traffic was sent to the sample HTTP server.
• source_metric: skipped.
• keda_metric: external.metrics.k8s.io returned a value at least equal to the exported pass threshold for a metric from ScaledObject.status.externalMetricNames.
• hpa_metric: the selected pod autoscaler’s status.currentMetrics included the KEDA external metric at a value at least equal to the exported pass threshold.
• hpa_active: the selected pod autoscaler reported ScalingActive=True.
• scale_up and scale_down: target Deployment replicas crossed configured thresholds.

Prometheus:

• signal: /work traffic was sent to work-simulator.
• source_metric: the Prometheus query returned a value at least equal to the configured and exported pass threshold.
• keda_metric: external.metrics.k8s.io returned the generated KEDA metric at a value at least equal to the exported pass threshold.
• hpa_metric: the selected pod autoscaler’s status.currentMetrics included the KEDA external metric at a value at least equal to the exported pass threshold.
• hpa_active, scale_up, and scale_down: same as HTTP.

OTEL:

• signal: a synthetic OTLP gauge was exported to the Kedify OTEL add-on.
• source_metric: the add-on REST query API returned a value at least equal to the configured and exported pass threshold from its metric store.
• keda_metric: external.metrics.k8s.io returned the generated KEDA metric at a value at least equal to the exported pass threshold.
• hpa_metric: the selected pod autoscaler’s status.currentMetrics included the KEDA external metric at a value at least equal to the exported pass threshold.
• hpa_active, scale_up, and scale_down: same as HTTP.

CPU:

• signal: sample load-generator CPU profile was activated.
• source_metric: metrics.k8s.io reported target Pod CPU at least equal to the configured and exported pass threshold.
• keda_metric: skipped.
• hpa_metric: the selected pod autoscaler’s status.currentMetrics included CPU resource metrics at a value at least equal to the exported pass threshold.
• hpa_active, scale_up, and scale_down: same as HTTP.

Memory:

• signal: sample load-generator memory profile was activated.
• source_metric: metrics.k8s.io reported target Pod memory at least equal to the configured and exported pass threshold.
• keda_metric: skipped.
• hpa_metric: the selected pod autoscaler’s status.currentMetrics included memory resource metrics at a value at least equal to the exported pass threshold.
• hpa_active, scale_up, and scale_down: same as HTTP.

PRA metrics

PRA publishes internal Prometheus metrics (names prefixed with kedify_agent_pra_), including:

• kedify_agent_pra_reconcile_count
• kedify_agent_pra_resource_updates
• kedify_agent_pra_total
• kedify_agent_pra_error_total
• kedify_agent_pra_scaling_active

Additional decision and sizing metrics are also exported:

• kedify_agent_pra_scale_decisions_total
• kedify_agent_pra_scale_blocked_total
• kedify_agent_pra_scale_delta_absolute
• kedify_agent_pra_scale_delta_percent
• kedify_agent_pra_evaluation_duration_seconds
• kedify_agent_pra_time_to_apply_seconds

Notes:

• kedify_agent_pra_scaling_active is derived from lastScaleTime + cooldown window.
• Delta histogram units: CPU absolute delta is measured in millicores; memory absolute delta is measured in MiB.

HTTP metric keys

Metrics are keyed by namespace/httpscaledobject-name. This means:

• All traffic matching an HTTPScaledObject is aggregated together
• Different paths, hosts, or query parameters within the same HSO share metrics

HTTP scaling metrics

RPS (Requests Per Second)

• Source: cluster.upstream_rq_total from Envoy
• Use case: scalingMetric: requestRate

Concurrency

• Source: cluster.upstream_rq_active from Envoy
• Use case: scalingMetric: concurrency