HTTP scaling for inference

Scale an inference Deployment from HTTP request rate or concurrency, and route each request through an Endpoint Picker selected by an InferencePool.

Details

Before configuring the scaler, deploy the model, an InferencePool, its Endpoint Picker, and a Service whose selector matches the pool. GPU capacity and model readiness are independent prerequisites. The Ingress inference tutorial installs the example dependencies and verifies requests end to end.

Architecture

Scroll to explore

Diagram description

Ingress requests reach Kedify Proxy, which exchanges ext_proc messages with the Endpoint Picker to select an InferencePool endpoint. The proxy forwards request traffic to the model backend, using the interceptor while activating from zero. HTTP statistics feed the external scaler, and KEDA with HPA or KPA scales the vLLM Deployment.

Trigger Specification

See HTTP scaler and inference configuration for the trigger fields.

Minimal routing configuration

With an existing InferencePool, Endpoint Picker, matching Service and Deployment, the trigger identifies the serving target:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: inference
  namespace: inference
spec:
  scaleTargetRef:
    name: inference
  minReplicaCount: 1
  maxReplicaCount: 3
  triggers:
    - type: kedify-http
      metadata:
        hosts: inference.example.com
        service: inference
        port: "8000"
        inferencePool: inference
        scalingMetric: concurrency
        targetValue: "5"
        trafficAutowire: ingress

Replace resource names and the target concurrency with values verified for your serving runtime. This example keeps one replica while you validate routing; it does not provision GPU nodes, models or Gateway extensions. Follow the complete Ingress inference procedure to install the example dependencies, generate traffic and verify ready serving capacity. Use inference diagnostics when a replica exists but cannot serve requests.

Traffic Autowiring

The example uses trafficAutowire: ingress. For other route types and GitOps ownership, see HTTP traffic autowiring. Verify that your installed version supports the chosen inference integration.

Inference traffic enters the namespace-local proxy on port 9002:

Ingress → kedify-proxy → Endpoint Picker (ext_proc) → kedify-proxy → model Pod

Autowiring Fallback

Fallback can restore the original route when the control plane is unhealthy; it still requires healthy serving replicas. See HTTP fallback behavior for timing and configuration. A direct Service route bypasses Endpoint Picker selection.

Disabling Traffic Autowiring

Set trafficAutowire: "false" only when you manage the route to the proxy yourself. Automatic route restoration does not apply to manually wired routes.

Kedify Proxy

Deployment Options for Kedify Proxy

Use namespace-local proxies for the Ingress example. General proxy placement and the agent.kedifyProxy.clusterWide Helm setting are described in HTTP scaling; cluster-wide support must not be inferred for every inference integration.

Kedify Proxy Traffic Flow

Envoy’s external processing filter asks the Endpoint Picker to select a backend. Routing policy belongs to that processor; the HTTP scaler adjusts Deployment replicas from traffic metrics.

External Processor

The external processor can be any service that implements the envoy ext_proc interfaces. Some common implementations for inference workloads are

• Gateway API Inference Extension Endpoint Picker (GAIE EPP)
• LLM-D Inference Scheduler
• vLLM Semantic Router

Scale-to-Zero for Inference Workloads

The HTTP Scaler fully supports scale-to-zero for inference workloads. When traffic stops flowing to your inference endpoint, KEDA will scale your deployment down to zero replicas after the configured cooldownPeriod.

How It Works

1. Metrics Collection: The kedify-proxy collects RPS (requests per second) and Concurrency metrics for all inference traffic on port 9002
2. Metric Aggregation: Metrics are pushed to the interceptor and forwarded to the scaler via gRPC bridge
3. Scaler Reports to KEDA: The scaler receives metrics and reports them to KEDA
4. Scaling Decision: KEDA uses these metrics to determine replica count, including scaling to zero when metrics reach zero

Enabling Scale-to-Zero

To enable scale-to-zero for inference workloads, set minReplicaCount: 0 in your ScaledObject.

Using the example from HTTP Scaling for Ingress-Based Inference Workloads, change the minReplicaCount from 1 to 0:

scaledobject.yaml

kind: ScaledObject
apiVersion: keda.sh/v1alpha1
metadata:
  name: inferencepool
  namespace: inferencepool
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llm-d-modelservice-decode
  cooldownPeriod: 600
  minReplicaCount: 0 # Changed from 1 to enable scale-to-zero
  maxReplicaCount: 2
  # ... rest of configuration from the how-to guide

Tip: For a complete working example including the inference scheduler (EPP) and model deployment, follow the HTTP Scaling for Ingress-Based Inference Workloads tutorial and modify minReplicaCount: 0.

Traffic Handling During Scale-to-Zero

When pods are scaled to zero, incoming requests are queued by the interceptor until pods become available. The traffic flow during cold start:

1. Request arrives at kedify-proxy (port 9002)
2. Interceptor detects no available pods and triggers scale-up
3. Request is held until pods are ready
4. Once ready, request flows through: kedify-proxy -> ext_proc (EPP) -> kedify-proxy -> Pod

Note: Consider setting an appropriate cooldownPeriod for inference workloads. Inference models often have longer startup times, so a longer cooldown (e.g., 300-600 seconds) prevents unnecessary scaling oscillations.

Verifying Inference Metrics

You can verify that metrics are being collected correctly for your inference workload using the following commands.

Check Current Metrics

Query the /queue endpoint to see real-time metrics:

kubectl get --raw /api/v1/namespaces/keda/services/keda-add-ons-http-interceptor-admin:9090/proxy/queue

Example output:

{
  "inferencepool/my-inference-hso": {
    "Concurrency": 5,
    "RPS": 42.5
  }
}

The key format is namespace/httpscaledobject-name.

Understanding the Metrics

Metric	Description	Use Case
RPS	Requests per second (time-windowed average)	Use with scalingMetric: requestRate
Concurrency	Current number of active/pending requests	Use with scalingMetric: concurrency

Testing with Traffic

Generate test traffic to verify metrics flow:

# Send 100 requests with 10 concurrent connections
hey -n 100 -c 10 -m POST \
  -host "inference.example.com" \
  -d '{"model": "my-model", "prompt": "test"}' \
  http://localhost:9080/v1/completions

Then check the metrics:

kubectl get --raw /api/v1/namespaces/keda/services/keda-add-ons-http-interceptor-admin:9090/proxy/queue | jq

Metrics Granularity

Metrics are aggregated per HTTPScaledObject, not per path or model. This means:

• All model requests (model-a, model-b) on the same HTTPScaledObject are combined

For independent models or paths, configure separate ScaledObjects with distinct serving targets; let the Agent manage their HTTPScaledObjects.

Troubleshooting

If metrics are not appearing:

1. Check the envoy-metrics-map: Verify the cluster-to-HSO mapping

   kubectl get --raw /api/v1/namespaces/keda/services/keda-add-ons-http-interceptor-admin:9090/proxy/envoy-metrics-map

2. Verify kedify-proxy is running: Ensure the proxy is deployed in your namespace

   kubectl get pods -n <your-namespace> | grep kedify-proxy

3. Check interceptor logs: Look for metrics-related messages

   kubectl logs -n keda -l app=keda-add-ons-http-interceptor -f

Continue with this topic

Configure and verify: Scale vLLM with OTel model metrics.

Diagnose: Inference pods are pending or requests are slow.