# Inference pods are pending or requests are slow

Inference failures can come from the request path, model initialization or node/device capacity. Changing replica targets only addresses some of them.

## Evidence, remedy and recovery check

### Requests fail before reaching a model

For the Ingress-based tutorial, inspect the routing resources in the workload namespace:

```bash
NS=my-inference
kubectl get ingress,service,endpointslices -n "$NS"
kubectl get httpscaledobjects -n "$NS"
kubectl get inferencepools.inference.networking.k8s.io -n "$NS" -o yaml
```

Expect the Ingress backend to resolve to the intended Service and the InferencePool selector to match serving pods. Inspect the endpoint-picker Service and its ready endpoints. An unknown InferencePool resource type indicates missing inference-extension CRDs. For Gateway-based routing, inspect HTTPRoutes and their parent acceptance instead.

Compare the [Ingress inference tutorial](https://docs.kedify.io/how-to/http-scaling-for-ingress-based-inference-workloads/) and [HTTP error logs](https://docs.kedify.io/how-to/kedify-proxy-access-logs-and-metrics/); test one request through each hop.

### Pods are Pending

Set `POD` to a pending model-serving pod:

```bash
POD=my-model-pod
kubectl describe pod "$POD" -n "$NS"
kubectl get events -n "$NS" --field-selector "involvedObject.name=$POD" --sort-by=.metadata.creationTimestamp
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, allocatable: .status.allocatable}'
```

`FailedScheduling` events identify insufficient devices, affinity/taint conflicts or volume constraints. Compare the exact requested GPU resource key with node allocatable resources; a Ready CPU-only node cannot satisfy a GPU request.

Resolve the provider/device constraint. GKE **standby** buffers exclude GPU/TPU nodes; active buffers have separate [provider requirements](https://docs.kedify.io/how-to/gke-capacity-buffers/#prerequisites). The example buffer does not provision GPU capacity.

### Pods run but are not ready

Inspect readiness and model-server logs:

```bash
kubectl get pod "$POD" -n "$NS" -o json | jq '{conditions: .status.conditions, containers: .status.containerStatuses}'
kubectl logs "$POD" -n "$NS" --all-containers=true --tail=100
```

Expect successful model loading and a passing readiness probe. Image-pull failures, download errors, `OOMKilled` or failed probes need different corrections; additional replicas do not fix them. Once the server is ready, repeat the [tutorial’s model-list and completion requests](https://docs.kedify.io/how-to/http-scaling-for-ingress-based-inference-workloads/).

Wait for model-ready evidence and test inference. Increase startup budgets only after identifying the delay.

### Model metrics are absent or select the wrong model

Inspect exporter/collector and metric labels for the serving runtime.

Use the [vLLM OTel tutorial](https://docs.kedify.io/how-to/otel-scaler-general/) and verify the query before changing scaling thresholds.

### Latency worsens on scale-in or from zero

Separate queue time, time to first token, generation time and stream duration. Check termination/draining and timeout budgets.

Keep a measured warm floor when cold starts miss the objective; evaluate [zero versus warm capacity](https://docs.kedify.io/guides/scale-to-zero/).

## Verify recovery

Repeat the same model request through the production route. Check successful responses, model-ready pods and time to first token; test cold starts separately from steady traffic.

For unresolved failures, [Collect diagnostics for support](https://docs.kedify.io/troubleshooting/collecting-kedify-configuration/) with versions, namespace/target, timestamps, relevant conditions and a redacted configuration.

Related: [Scale inference behind an Ingress](https://docs.kedify.io/how-to/http-scaling-for-ingress-based-inference-workloads/) · [Scale vLLM with OTel model metrics](https://docs.kedify.io/how-to/otel-scaler-general/) · [Diagnose HTTP requests with logs and error metrics](https://docs.kedify.io/how-to/kedify-proxy-access-logs-and-metrics/) · [Workload does not scale, or scales too slowly](https://docs.kedify.io/troubleshooting/workload-scaling/) · [HTTP inference scaling](https://docs.kedify.io/scalers/http-scaler-for-inference/).

---
Canonical: https://docs.kedify.io/troubleshooting/inference/
Source: src/content/docs/troubleshooting/inference.md
Documentation index: https://docs.kedify.io/llms.txt