# HTTP scaling for inference

Scale an inference Deployment from HTTP request rate or concurrency, and route each request through an Endpoint Picker selected by an `InferencePool`.

## Details

Before configuring the scaler, deploy the model, an `InferencePool`, its Endpoint Picker, and a Service whose selector matches the pool. GPU capacity and model readiness are independent prerequisites. The [Ingress inference tutorial](https://docs.kedify.io/how-to/http-scaling-for-ingress-based-inference-workloads/) installs the example dependencies and verifies requests end to end.

## Architecture

[![HTTP Scaler Architecture](https://docs.kedify.io/assets/images/docs/http-scaler-inference.svg)](https://docs.kedify.io/assets/images/docs/http-scaler-inference.svg)

Scroll to exploreDiagram description

Ingress requests reach Kedify Proxy, which exchanges ext_proc messages with the Endpoint Picker to select an InferencePool endpoint. The proxy forwards request traffic to the model backend, using the interceptor while activating from zero. HTTP statistics feed the external scaler, and KEDA with HPA or KPA scales the vLLM Deployment.

## Trigger Specification

See [HTTP scaler and inference configuration](https://docs.kedify.io/reference/http-scaler/) for the trigger fields.

## Minimal routing configuration

With an existing `InferencePool`, Endpoint Picker, matching Service and Deployment, the trigger identifies the serving target:

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: inference
  namespace: inference
spec:
  scaleTargetRef:
    name: inference
  minReplicaCount: 1
  maxReplicaCount: 3
  triggers:
    - type: kedify-http
      metadata:
        hosts: inference.example.com
        service: inference
        port: "8000"
        inferencePool: inference
        scalingMetric: concurrency
        targetValue: "5"
        trafficAutowire: ingress
```

Replace resource names and the target concurrency with values verified for your serving runtime. This example keeps one replica while you validate routing; it does not provision GPU nodes, models or Gateway extensions. Follow the [complete Ingress inference procedure](https://docs.kedify.io/how-to/http-scaling-for-ingress-based-inference-workloads/) to install the example dependencies, generate traffic and verify ready serving capacity. Use [inference diagnostics](https://docs.kedify.io/troubleshooting/inference/) when a replica exists but cannot serve requests.

## Traffic Autowiring

The example uses `trafficAutowire: ingress`. For other route types and GitOps ownership, see [HTTP traffic autowiring](https://docs.kedify.io/scalers/http-scaler/#traffic-autowiring). Verify that your installed version supports the chosen inference integration.

Inference traffic enters the namespace-local proxy on port **9002**:

`Ingress → kedify-proxy → Endpoint Picker (ext_proc) → kedify-proxy → model Pod`

### Autowiring Fallback

Fallback can restore the original route when the control plane is unhealthy; it still requires healthy serving replicas. See [HTTP fallback behavior](https://docs.kedify.io/scalers/http-scaler/#autowiring-fallback) for timing and configuration. A direct Service route bypasses Endpoint Picker selection.

### Disabling Traffic Autowiring

Set `trafficAutowire: "false"` only when you manage the route to the proxy yourself. Automatic route restoration does not apply to manually wired routes.

## Kedify Proxy

### Deployment Options for Kedify Proxy

Use namespace-local proxies for the Ingress example. General proxy placement and the `agent.kedifyProxy.clusterWide` Helm setting are described in [HTTP scaling](https://docs.kedify.io/scalers/http-scaler/#deployment-options-for-kedify-proxy); cluster-wide support must not be inferred for every inference integration.

### Kedify Proxy Traffic Flow

Envoy’s [external processing filter](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/ext_proc/v3/ext_proc.proto#envoy-v3-api-msg-extensions-filters-http-ext-proc-v3-externalprocessor) asks the Endpoint Picker to select a backend. Routing policy belongs to that processor; the HTTP scaler adjusts Deployment replicas from traffic metrics.

### External Processor

The external processor can be any service that implements the envoy `ext_proc` interfaces. Some common implementations for inference workloads are

- [Gateway API Inference Extension Endpoint Picker (GAIE EPP)](https://gateway-api-inference-extension.sigs.k8s.io/guides/implementers/)

- [LLM-D Inference Scheduler](https://github.com/llm-d/llm-d-inference-scheduler)

- [vLLM Semantic Router](https://github.com/vllm-project/semantic-router)

## Scale-to-Zero for Inference Workloads

The HTTP Scaler fully supports scale-to-zero for inference workloads. When traffic stops flowing to your inference endpoint, KEDA will scale your deployment down to zero replicas after the configured `cooldownPeriod`.

### How It Works

1. **Metrics Collection**: The `kedify-proxy` collects RPS (requests per second) and Concurrency metrics for all inference traffic on port 9002

2. **Metric Aggregation**: Metrics are pushed to the interceptor and forwarded to the scaler via gRPC bridge

3. **Scaler Reports to KEDA**: The scaler receives metrics and reports them to KEDA

4. **Scaling Decision**: KEDA uses these metrics to determine replica count, including scaling to zero when metrics reach zero

### Enabling Scale-to-Zero

To enable scale-to-zero for inference workloads, set `minReplicaCount: 0` in your ScaledObject.

Using the example from [HTTP Scaling for Ingress-Based Inference Workloads](https://docs.kedify.io/how-to/http-scaling-for-ingress-based-inference-workloads/), change the `minReplicaCount` from `1` to `0`:

scaledobject.yaml

```yaml
kind: ScaledObject
apiVersion: keda.sh/v1alpha1
metadata:
  name: inferencepool
  namespace: inferencepool
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llm-d-modelservice-decode
  cooldownPeriod: 600
  minReplicaCount: 0 # Changed from 1 to enable scale-to-zero
  maxReplicaCount: 2
  # ... rest of configuration from the how-to guide
```

**Tip:** For a complete working example including the inference scheduler (EPP) and model deployment, follow the [HTTP Scaling for Ingress-Based Inference Workloads](https://docs.kedify.io/how-to/http-scaling-for-ingress-based-inference-workloads/) tutorial and modify `minReplicaCount: 0`.

### Traffic Handling During Scale-to-Zero

When pods are scaled to zero, incoming requests are queued by the interceptor until pods become available. The traffic flow during cold start:

1. Request arrives at `kedify-proxy` (port 9002)

2. Interceptor detects no available pods and triggers scale-up

3. Request is held until pods are ready

4. Once ready, request flows through: `kedify-proxy -> ext_proc (EPP) -> kedify-proxy -> Pod`

**Note:** Consider setting an appropriate `cooldownPeriod` for inference workloads. Inference models often have longer startup times, so a longer cooldown (e.g., 300-600 seconds) prevents unnecessary scaling oscillations.

## Verifying Inference Metrics

You can verify that metrics are being collected correctly for your inference workload using the following commands.

### Check Current Metrics

Query the `/queue` endpoint to see real-time metrics:

```bash
kubectl get --raw /api/v1/namespaces/keda/services/keda-add-ons-http-interceptor-admin:9090/proxy/queue
```

Example output:

```json
{
  "inferencepool/my-inference-hso": {
    "Concurrency": 5,
    "RPS": 42.5
  }
}
```

The key format is `namespace/httpscaledobject-name`.

### Understanding the Metrics

| Metric | Description | Use Case |
| --- | --- | --- |
| **RPS** | Requests per second (time-windowed average) | Use with `scalingMetric: requestRate` |
| **Concurrency** | Current number of active/pending requests | Use with `scalingMetric: concurrency` |

### Testing with Traffic

Generate test traffic to verify metrics flow:

```bash
# Send 100 requests with 10 concurrent connections
hey -n 100 -c 10 -m POST \
  -host "inference.example.com" \
  -d '{"model": "my-model", "prompt": "test"}' \
  http://localhost:9080/v1/completions
```

Then check the metrics:

```bash
kubectl get --raw /api/v1/namespaces/keda/services/keda-add-ons-http-interceptor-admin:9090/proxy/queue | jq
```

### Metrics Granularity

Metrics are aggregated **per HTTPScaledObject**, not per path or model. This means:

- All model requests (model-a, model-b) on the same HTTPScaledObject are combined

For independent models or paths, configure separate ScaledObjects with distinct serving targets; let the Agent manage their HTTPScaledObjects.

### Troubleshooting

If metrics are not appearing:

1. **Check the envoy-metrics-map**: Verify the cluster-to-HSO mapping

    

   ```bash
kubectl get --raw /api/v1/namespaces/keda/services/keda-add-ons-http-interceptor-admin:9090/proxy/envoy-metrics-map
```

2. **Verify kedify-proxy is running**: Ensure the proxy is deployed in your namespace

    

   ```bash
kubectl get pods -n <your-namespace> | grep kedify-proxy
```

3. **Check interceptor logs**: Look for metrics-related messages

    

   ```bash
kubectl logs -n keda -l app=keda-add-ons-http-interceptor -f
```

## Continue with this topic

**Configure and verify:** [Scale vLLM with OTel model metrics](https://docs.kedify.io/how-to/otel-scaler-general/).

**Diagnose:** [Inference pods are pending or requests are slow](https://docs.kedify.io/troubleshooting/inference/).

Last updated: Sep 21, 2026

---
Canonical: https://docs.kedify.io/scalers/http-scaler-for-inference/
Source: src/content/docs/scalers/http-scaler-for-inference.md
Documentation index: https://docs.kedify.io/llms.txt