# Diagnose HTTP requests with logs and error metrics

The `kedify-http` scaler routes application traffic through `kedify-proxy`, a fleet of Envoy proxies. When a client receives an error response, for example a 404, it can come from two different places:

1. **The application** returned it. The request was proxied to an application pod and the application decided to answer 404.

2. **The proxy** returned it. The request did not match any known host or route, so Envoy answered 404 itself without contacting any application.

Telling these two apart is the first step when troubleshooting unexpected errors. This guide shows how to do that with Envoy metrics and how to enable access logs for per-request detail.

## Error metrics

Every `kedify-proxy` pod exposes the standard [Envoy admin interface](https://www.envoyproxy.io/docs/envoy/latest/operations/admin) on port `9901`, including Prometheus metrics at `/stats/prometheus`. The `kedify-proxy` helm chart also creates a `kedify-proxy-admin` service for it by default (value `service.exposeAdminInterface`), so the metrics can be scraped with a `ServiceMonitor`:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kedify-proxy
spec:
  endpoints:
    - port: admin
      scheme: http
      path: /stats/prometheus
  namespaceSelector:
    matchNames:
      - my-app-namespace
  selector:
    matchLabels:
      app: kedify-proxy
```

Three metrics answer the “who returned the error” question:

### 404 returned by the application

```plaintext
envoy_cluster_upstream_rq{envoy_response_code="404", envoy_cluster_name="<namespace>/<httpscaledobject-name>"}
```

Envoy counts every response code it receives from an upstream, per cluster. Kedify names each Envoy cluster after the `HTTPScaledObject` it belongs to, in the `<namespace>/<name>` format, so the `envoy_cluster_name` label directly identifies the scaled application. There is also the aggregated `envoy_cluster_upstream_rq_xx{envoy_response_code_class="4"}` variant.

While an application is scaled to zero or has no ready endpoints, its cluster temporarily points to the Kedify interceptor, which holds the request until the application wakes up. Responses served during this cold start window are counted on the same cluster even though they were forwarded by the interceptor. If you need to tell the two cases apart, use access logs, where the `UpstreamHost` field shows who actually answered.

### 404 returned by the proxy

```plaintext
envoy_http_no_route{envoy_http_conn_manager_prefix="kedify-proxy"}
```

This counter increments every time a request does not match any configured virtual host or route, in which case Envoy answers 404 by itself. The `kedify-proxy` prefix covers both the plaintext and the TLS listener, so one query covers all traffic.

This metric intentionally has no hostname label. A request counted here did not match any configuration object, so there is nothing to attribute it to, and Envoy does not offer per-hostname statistics for unmatched requests. The metric is emitted per pod, which makes it useful for spotting a single proxy instance with an outdated route table. To see which hostnames were requested, enable access logs as described below.

### All 4xx sent to clients

```plaintext
envoy_http_downstream_rq_xx{envoy_response_code_class="4", envoy_http_conn_manager_prefix="kedify-proxy"}
```

This counts every 4xx response sent downstream, both proxied application responses and Envoy generated ones, and serves as a cross-check for the two metrics above.

## Access logs

Metrics tell you how many errors occurred; access logs tell you which host, path, and upstream each individual request had. Access logging is disabled by default and is configured through a single environment variable on the http-add-on **interceptor**, which acts as the xDS control plane for the whole `kedify-proxy` fleet:

| Value of `KEDIFY_PROXY_ACCESS_LOG_TYPE` | Behavior |
| --- | --- |
| *(unset or empty)* | Access logging disabled (default) |
| `json` | One JSON object per request |
| `plaintext` | One human-readable line per request |

Set it in your helm values:

```yaml
keda-add-ons-http:
  interceptor:
    extraEnvs:
      KEDIFY_PROXY_ACCESS_LOG_TYPE: json
```

Older versions of the http-add-on chart named this value `interceptor.additionalEnvVars` and expected a list of `name`/`value` pairs instead of a map. If `extraEnvs` does not appear in your chart version’s values, use the older form.

Or for a quick test on an existing installation:

```bash
kubectl set env -nkeda deploy/keda-add-ons-http-interceptor KEDIFY_PROXY_ACCESS_LOG_TYPE=json
```

The interceptor restarts to pick up the variable and then delivers the new listener configuration to all `kedify-proxy` pods over xDS, without restarting them. Each proxy writes the access log to its stdout, so the logs are collected the same way as any other pod logs.

Access logs are written for every request flowing through the proxies. Filtering by status code is currently not supported, so account for the log volume on busy clusters before enabling it fleet-wide.

### Reading the log

A proxy-generated 404 looks like this (fields shortened for readability):

```json
{
  "Timestamp": "2026-07-22T09:15:04.123",
  "Method": "GET",
  "Path": "/api/orders",
  "Protocol": "HTTP/1.1",
  "ResponseCode": "404",
  "ResponseFlags": "NR",
  "Duration": "0",
  "RequestId": "5c9b...",
  "Authority": "preview-42.example.com",
  "UpstreamHost": "-"
}
```

And an application-returned 404 like this:

```json
{
  "Timestamp": "2026-07-22T09:15:09.456",
  "Method": "GET",
  "Path": "/api/orders",
  "Protocol": "HTTP/1.1",
  "ResponseCode": "404",
  "ResponseFlags": "-",
  "Duration": "3",
  "RequestId": "81af...",
  "Authority": "preview-42.example.com",
  "UpstreamHost": "10.244.1.17:8080"
}
```

The fields that matter for attribution:

| Field | Meaning |
| --- | --- |
| `Authority` | The requested hostname. This is how you correlate proxy-generated 404s with a particular application or `HTTPScaledObject`. |
| `ResponseFlags` | Envoy’s reason for a response it generated itself. `NR` means no route matched the request. `-` means Envoy has nothing to report, i.e. the response came from the upstream. See the [Envoy response flags reference](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#config-access-log-format-response-flags) for the full list. |
| `UpstreamHost` | The address that answered the request. An application pod IP for regular traffic, `-` when Envoy answered itself, or the interceptor address during cold starts. |

In short: `ResponseFlags: "NR"` with no `UpstreamHost` is the proxy saying “I do not know this host or route”, typically because the routing configuration for that hostname does not exist or has not reached this proxy pod yet. Any other 404 with a pod IP in `UpstreamHost` was a decision made by your application.

## Related configuration

To further diagnose or mitigate proxy-generated 404s:

- `KEDIFY_ENVOY_ENABLE_FALLBACK_CATCH_ALL_ROUTE=true` on the interceptor adds a wildcard fallback route: instead of answering 404 for an unknown host, the proxy forwards the request to the interceptor, which holds the authoritative routing table. This both hides short configuration propagation delays from clients and makes the fallback traffic measurable via `envoy_cluster_upstream_rq{envoy_cluster_name="wildcard_fallback_catch_all_cluster"}`. Note that this option is incompatible with the `pathEmbeddedHost` health check annotation, because both use the wildcard domain.

- Advanced Envoy settings such as retries, health checks, or bootstrap overrides are covered in [Configure Envoy in the Kedify Proxy](https://docs.kedify.io/how-to/kedify-proxy-envoy-configuration/).

- General performance oriented monitoring of the proxy fleet is covered in [Performance Tuning](https://docs.kedify.io/how-to/http-proxy-performance-tuning/).

## Continue with this topic

**Reference:** [HTTP scaler and inference configuration](https://docs.kedify.io/reference/http-scaler/) · [Metrics, status and health signals](https://docs.kedify.io/reference/metrics/).

**Diagnose:** [Workload does not scale, or scales too slowly](https://docs.kedify.io/troubleshooting/workload-scaling/).

**Related capabilities:** [HTTP scaling](https://docs.kedify.io/scalers/http-scaler/) · [How HTTP scaling works](https://docs.kedify.io/concepts/http-scaler-metrics-architecture/).

---
Canonical: https://docs.kedify.io/how-to/kedify-proxy-access-logs-and-metrics/
Source: src/content/docs/how-to/kedify-proxy-access-logs-and-metrics.md
Documentation index: https://docs.kedify.io/llms.txt