Metrics, status and health signals

Metrics are emitted by specific installed components. Check units, labels and sample freshness before comparing control-plane health with application success. Use Workload diagnostics for the measurement workflow.

For endpoints and collection with Datadog, Prometheus and other stacks, see Monitoring.

Source and enablement

Verify the component’s /metrics output for the installed release. Missing telemetry does not by itself mean the scaling control loop failed.

Emitter	Evidence	Enablement / task
Check runners	End-to-end canary timing and step outcomes	Install and verify checks
HTTP proxy/interceptor	Request rate, concurrency, errors and route ownership	HTTP log/metric diagnosis
Agent PRA controller	Resource decisions, blocked changes, deltas and evaluation timing	Enable PRA
HPA/KPA and custom resources	Desired/current state and conditions	Kubernetes API catalog; exact status fields stay with each kind

Member-cluster CA expiry

The Agent exposes kedify_agent_kedifyconfiguration_multicluster_ca_expiration_timestamp_seconds for parseable CA certificates in member kubeconfigs. See multi-cluster monitoring for labels, days-remaining queries and expiry alerts.

Cardinality and interpretation

Keep namespace/workload/container dimensions bounded to the workloads you operate; do not add request IDs, error messages or timestamps as metric labels. Check-runner result labels are a bounded vocabulary. A fresh healthy canary does not establish that a business request, model or queue worker meets its objective. HTTP traces provide a separate request-timing view.

Every runner exposes /healthz and /metrics.

For Kedify Dashboard ingestion, the Kedify agent auto-discovers labeled runner Services and relays these metrics with kedify_agent_id, which the dashboard API uses to scope results to the selected cluster.

Primary Metric

kedify_autoscaling_check_duration_seconds{
  check="http",
  result="success",
  signal="success",
  source_metric="skipped",
  keda_metric="success",
  hpa_metric="success",
  hpa_active="success",
  scale_up="success",
  scale_down="success"
} 42.5

This gauge is the primary alerting metric. It records the duration of the latest complete check iteration, in seconds. Alert on this metric when autoscaling control-plane latency exceeds your threshold, for example:

kedify_autoscaling_check_duration_seconds > 300

Example Prometheus alert rule:

groups:
  - name: kedify-autoscaling-checks
    rules:
      - alert: KedifyAutoscalingCheckSlow
        expr: kedify_autoscaling_check_duration_seconds > 300
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: Kedify autoscaling check is slow
          description: Check {{ $labels.check }} took {{ $value }}s in the latest iteration.

The result and step labels explain what happened during that iteration. Label values are bounded to:

success | failure | skipped | unknown

Dynamic object names, timestamps, reasons, and error messages are not emitted as labels. Result labels are bounded; kedify_autoscaling_check_info is the static metadata exception and exposes bounded values such as the runner namespace, primary metric name, and interval.

Supporting Metrics

kedify_autoscaling_check_step_duration_seconds{check="http",step="keda_metric"}
kedify_autoscaling_check_last_run_timestamp_seconds{check="http"}
kedify_autoscaling_check_next_run_timestamp_seconds{check="http"}
kedify_autoscaling_check_info{check="http",runner_namespace="keda",primary_metric="kedify_autoscaling_check_duration_seconds",interval_seconds="300",autoscaler_class="hpa"}
kedify_autoscaling_check_metric_threshold{check="prometheus",metric="source"}
kedify_autoscaling_check_observed_metric_value{check="prometheus",metric="source"}
kedify_autoscaling_check_observed_metric_value{check="otel",metric="keda"}
kedify_autoscaling_check_observed_replicas{check="cpu",phase="scale_up"}
kedify_autoscaling_check_observed_replicas{check="memory",phase="scale_up"}

Metric types:

• kedify_autoscaling_check_duration_seconds: gauge, latest complete iteration duration.
• kedify_autoscaling_check_step_duration_seconds: gauge, latest step duration.
• kedify_autoscaling_check_last_run_timestamp_seconds: gauge, Unix timestamp of the latest iteration.
• kedify_autoscaling_check_next_run_timestamp_seconds: gauge, Unix timestamp of the next scheduled iteration.
• kedify_autoscaling_check_info: gauge, static runner metadata for the check.
• kedify_autoscaling_check_metric_threshold: gauge, minimum value used to decide whether the metric check passes.
• kedify_autoscaling_check_observed_metric_value: gauge, observed source, KEDA, or selected pod-autoscaler metric value.
• kedify_autoscaling_check_observed_replicas: gauge, observed target replica count.

Metric threshold and observed value labels use static names such as:

• source
• keda
• hpa

The hpa source and the hpa_metric/hpa_active result labels are retained for dashboard and alert compatibility. When autoscaler.class=kpa, they represent the HPA-compatible KPA status read directly from KedifyPodAutoscaler.

kedify_autoscaling_check_info includes autoscaler_class="hpa|kpa" so consumers can distinguish which control-plane path each runner validates without changing the compatibility step labels.

Replica phases use:

• scale_up
• scale_down

Check Semantics

HTTP:

• signal: in-cluster HTTP traffic was sent to the sample HTTP server.
• source_metric: skipped.
• keda_metric: external.metrics.k8s.io returned a value at least equal to the exported pass threshold for a metric from ScaledObject.status.externalMetricNames.
• hpa_metric: the selected pod autoscaler’s status.currentMetrics included the KEDA external metric at a value at least equal to the exported pass threshold.
• hpa_active: the selected pod autoscaler reported ScalingActive=True.
• scale_up and scale_down: target Deployment replicas crossed configured thresholds.

Prometheus:

• signal: /work traffic was sent to work-simulator.
• source_metric: the Prometheus query returned a value at least equal to the configured and exported pass threshold.
• keda_metric: external.metrics.k8s.io returned the generated KEDA metric at a value at least equal to the exported pass threshold.
• hpa_metric: the selected pod autoscaler’s status.currentMetrics included the KEDA external metric at a value at least equal to the exported pass threshold.
• hpa_active, scale_up, and scale_down: same as HTTP.

OTEL:

• signal: a synthetic OTLP gauge was exported to the Kedify OTEL add-on.
• source_metric: the add-on REST query API returned a value at least equal to the configured and exported pass threshold from its metric store.
• keda_metric: external.metrics.k8s.io returned the generated KEDA metric at a value at least equal to the exported pass threshold.
• hpa_metric: the selected pod autoscaler’s status.currentMetrics included the KEDA external metric at a value at least equal to the exported pass threshold.
• hpa_active, scale_up, and scale_down: same as HTTP.

CPU:

• signal: sample load-generator CPU profile was activated.
• source_metric: metrics.k8s.io reported target Pod CPU at least equal to the configured and exported pass threshold.
• keda_metric: skipped.
• hpa_metric: the selected pod autoscaler’s status.currentMetrics included CPU resource metrics at a value at least equal to the exported pass threshold.
• hpa_active, scale_up, and scale_down: same as HTTP.

Memory:

• signal: sample load-generator memory profile was activated.
• source_metric: metrics.k8s.io reported target Pod memory at least equal to the configured and exported pass threshold.
• keda_metric: skipped.
• hpa_metric: the selected pod autoscaler’s status.currentMetrics included memory resource metrics at a value at least equal to the exported pass threshold.
• hpa_active, scale_up, and scale_down: same as HTTP.

PRA metrics

PRA publishes internal Prometheus metrics (names prefixed with kedify_agent_pra_), including:

• kedify_agent_pra_reconcile_count
• kedify_agent_pra_resource_updates
• kedify_agent_pra_total
• kedify_agent_pra_error_total
• kedify_agent_pra_scaling_active

Additional decision and sizing metrics are also exported:

• kedify_agent_pra_scale_decisions_total
• kedify_agent_pra_scale_blocked_total
• kedify_agent_pra_scale_delta_absolute
• kedify_agent_pra_scale_delta_percent
• kedify_agent_pra_evaluation_duration_seconds
• kedify_agent_pra_time_to_apply_seconds

Notes:

• kedify_agent_pra_scaling_active is derived from lastScaleTime + cooldown window.
• Delta histogram units: CPU absolute delta is measured in millicores; memory absolute delta is measured in MiB.

Datadog Agent metric selectors

Use these selectors in the metrics list of the Datadog OpenMetrics configuration. For Datadog Agent 7.32.0+ using openmetrics_endpoint, counter selectors omit the exported _total suffix, and submitted names gain .count. Namespace and alias determine the rest of the submitted name; they do not change the selector.

The following is the complete list of Kedify-defined counters ending in _total emitted by Kedify Agent v0.8.3. Other installed components and Go/controller-runtime metrics are outside this list. A series may appear only when its feature is enabled and the relevant event occurs. Verify the installed release’s /metrics output when upgrading.

Exported counter	Datadog selector
kedify_agent_pra_scale_blocked_total	kedify_agent_pra_scale_blocked
kedify_agent_pra_scale_decisions_total	kedify_agent_pra_scale_decisions
kedify_agent_distributedscaledjob_job_creations_total	kedify_agent_distributedscaledjob_job_creations
kedify_agent_vcluster_bootstrap_reconciliations_total	kedify_agent_vcluster_bootstrap_reconciliations
kedify_sharding_assignment_total	kedify_sharding_assignment
kedify_sharding_fallback_total	kedify_sharding_fallback

For example, with namespace: kedify.agent.pra, this entry preserves an existing scale_blocked_total alias:

{ "kedify_agent_pra_scale_blocked": "scale_blocked_total" }

The resulting Datadog metric is kedify.agent.pra.scale_blocked_total.count.

Names that stay unchanged

These Agent v0.8.3 metrics are gauges, despite ending in _total. Keep the full name as the selector; the counter .count suffix does not apply:

• kedify_agent_pra_error_total
• kedify_agent_pra_total
• kedify_agent_prp_error_total
• kedify_agent_prp_total
• kedify_agent_scaling_policy_total

The Agent’s other Kedify-defined counters already have selectors without _total: kedify_agent_grpc_msg_sent_count, kedify_agent_grpc_msg_received_count, kedify_agent_prp_reconcile_count, kedify_agent_prp_resource_updates, kedify_agent_pra_reconcile_count and kedify_agent_pra_resource_updates. Keep those selectors unchanged; Datadog still appends .count to their submitted aliases. PRA histogram selectors, such as kedify_agent_pra_evaluation_duration_seconds, also remain unchanged; their generated series follow Datadog’s histogram mapping.

If a metric is visible at the endpoint but absent in Datadog, inspect its # TYPE declaration and selector, then run agent check openmetrics in the Datadog Agent scraping that pod. Check the submitted name after namespace, alias and type suffix are applied. See Datadog’s OpenMetrics reference for collection and validation details.

HTTP metric keys

Metrics are keyed by namespace/httpscaledobject-name. This means:

• All traffic matching an HTTPScaledObject is aggregated together
• Different paths, hosts, or query parameters within the same HSO share metrics

HTTP scaling metrics

RPS (Requests Per Second)

• Source: cluster.upstream_rq_total from Envoy
• Use case: scalingMetric: requestRate

Concurrency

• Source: cluster.upstream_rq_active from Envoy
• Use case: scalingMetric: concurrency