Access Logs and Error Metrics in Kedify Proxy

The kedify-http scaler routes application traffic through kedify-proxy, a fleet of Envoy proxies. When a client receives an error response, for example a 404, it can come from two different places:

1. The application returned it. The request was proxied to an application pod and the application decided to answer 404.
2. The proxy returned it. The request did not match any known host or route, so Envoy answered 404 itself without contacting any application.

Telling these two apart is the first step when troubleshooting unexpected errors. This guide shows how to do that with Envoy metrics and how to enable access logs for per-request detail.

Error metrics

Every kedify-proxy pod exposes the standard Envoy admin interface on port 9901, including Prometheus metrics at /stats/prometheus. The kedify-proxy helm chart also creates a kedify-proxy-admin service for it by default (value service.exposeAdminInterface), so the metrics can be scraped with a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kedify-proxy
spec:
  endpoints:
    - port: admin
      scheme: http
      path: /stats/prometheus
  namespaceSelector:
    matchNames:
      - my-app-namespace
  selector:
    matchLabels:
      app: kedify-proxy

Three metrics answer the “who returned the error” question:

404 returned by the application

envoy_cluster_upstream_rq{envoy_response_code="404", envoy_cluster_name="<namespace>/<httpscaledobject-name>"}

Envoy counts every response code it receives from an upstream, per cluster. Kedify names each Envoy cluster after the HTTPScaledObject it belongs to, in the <namespace>/<name> format, so the envoy_cluster_name label directly identifies the scaled application. There is also the aggregated envoy_cluster_upstream_rq_xx{envoy_response_code_class="4"} variant.

Note

While an application is scaled to zero or has no ready endpoints, its cluster temporarily points to the Kedify interceptor, which holds the request until the application wakes up. Responses served during this cold start window are counted on the same cluster even though they were forwarded by the interceptor. If you need to tell the two cases apart, use access logs, where the UpstreamHost field shows who actually answered.

404 returned by the proxy

envoy_http_no_route{envoy_http_conn_manager_prefix="kedify-proxy"}

This counter increments every time a request does not match any configured virtual host or route, in which case Envoy answers 404 by itself. The kedify-proxy prefix covers both the plaintext and the TLS listener, so one query covers all traffic.

This metric intentionally has no hostname label. A request counted here did not match any configuration object, so there is nothing to attribute it to, and Envoy does not offer per-hostname statistics for unmatched requests. The metric is emitted per pod, which makes it useful for spotting a single proxy instance with an outdated route table. To see which hostnames were requested, enable access logs as described below.

All 4xx sent to clients

envoy_http_downstream_rq_xx{envoy_response_code_class="4", envoy_http_conn_manager_prefix="kedify-proxy"}

This counts every 4xx response sent downstream, both proxied application responses and Envoy generated ones, and serves as a cross-check for the two metrics above.

Access logs

Metrics tell you how many errors occurred; access logs tell you which host, path, and upstream each individual request had. Access logging is disabled by default and is configured through a single environment variable on the http-add-on interceptor, which acts as the xDS control plane for the whole kedify-proxy fleet:

Value of KEDIFY_PROXY_ACCESS_LOG_TYPE	Behavior
(unset or empty)	Access logging disabled (default)
json	One JSON object per request
plaintext	One human-readable line per request

Set it in your helm values:

keda-add-ons-http:
  interceptor:
    extraEnvs:
      KEDIFY_PROXY_ACCESS_LOG_TYPE: json

Note

Older versions of the http-add-on chart named this value interceptor.additionalEnvVars and expected a list of name/value pairs instead of a map. If extraEnvs does not appear in your chart version’s values, use the older form.

Or for a quick test on an existing installation:

kubectl set env -nkeda deploy/keda-add-ons-http-interceptor KEDIFY_PROXY_ACCESS_LOG_TYPE=json

The interceptor restarts to pick up the variable and then delivers the new listener configuration to all kedify-proxy pods over xDS, without restarting them. Each proxy writes the access log to its stdout, so the logs are collected the same way as any other pod logs.

Caution

Access logs are written for every request flowing through the proxies. Filtering by status code is currently not supported, so account for the log volume on busy clusters before enabling it fleet-wide.

Reading the log

A proxy-generated 404 looks like this (fields shortened for readability):

{"Timestamp":"2026-07-22T09:15:04.123","Method":"GET","Path":"/api/orders","Protocol":"HTTP/1.1","ResponseCode":"404","ResponseFlags":"NR","Duration":"0","RequestId":"5c9b...","Authority":"preview-42.example.com","UpstreamHost":"-"}

And an application-returned 404 like this:

{"Timestamp":"2026-07-22T09:15:09.456","Method":"GET","Path":"/api/orders","Protocol":"HTTP/1.1","ResponseCode":"404","ResponseFlags":"-","Duration":"3","RequestId":"81af...","Authority":"preview-42.example.com","UpstreamHost":"10.244.1.17:8080"}

The fields that matter for attribution:

Field	Meaning
Authority	The requested hostname. This is how you correlate proxy-generated 404s with a particular application or HTTPScaledObject.
ResponseFlags	Envoy’s reason for a response it generated itself. NR means no route matched the request. - means Envoy has nothing to report, i.e. the response came from the upstream. See the Envoy response flags reference for the full list.
UpstreamHost	The address that answered the request. An application pod IP for regular traffic, - when Envoy answered itself, or the interceptor address during cold starts.

In short: ResponseFlags: "NR" with no UpstreamHost is the proxy saying “I do not know this host or route”, typically because the routing configuration for that hostname does not exist or has not reached this proxy pod yet. Any other 404 with a pod IP in UpstreamHost was a decision made by your application.

Related configuration

To further diagnose or mitigate proxy-generated 404s:

• KEDIFY_ENVOY_ENABLE_FALLBACK_CATCH_ALL_ROUTE=true on the interceptor adds a wildcard fallback route: instead of answering 404 for an unknown host, the proxy forwards the request to the interceptor, which holds the authoritative routing table. This both hides short configuration propagation delays from clients and makes the fallback traffic measurable via envoy_cluster_upstream_rq{envoy_cluster_name="wildcard_fallback_catch_all_cluster"}. Note that this option is incompatible with the pathEmbeddedHost health check annotation, because both use the wildcard domain.
• Advanced Envoy settings such as retries, health checks, or bootstrap overrides are covered in Configure Envoy in the Kedify Proxy.
• General performance oriented monitoring of the proxy fleet is covered in Performance Tuning.