Inference pods are pending or requests are slow

Inference failures can come from the request path, model initialization or node/device capacity. Changing replica targets only addresses some of them.

Evidence, remedy and recovery check

Requests fail before reaching a model

For the Ingress-based tutorial, inspect the routing resources in the workload namespace:

NS=my-inference
kubectl get ingress,service,endpointslices -n "$NS"
kubectl get httpscaledobjects -n "$NS"
kubectl get inferencepools.inference.networking.k8s.io -n "$NS" -o yaml

Expect the Ingress backend to resolve to the intended Service and the InferencePool selector to match serving pods. Inspect the endpoint-picker Service and its ready endpoints. An unknown InferencePool resource type indicates missing inference-extension CRDs. For Gateway-based routing, inspect HTTPRoutes and their parent acceptance instead.

Compare the Ingress inference tutorial and HTTP error logs; test one request through each hop.

Pods are Pending

Set POD to a pending model-serving pod:

POD=my-model-pod
kubectl describe pod "$POD" -n "$NS"
kubectl get events -n "$NS" --field-selector "involvedObject.name=$POD" --sort-by=.metadata.creationTimestamp
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, allocatable: .status.allocatable}'

FailedScheduling events identify insufficient devices, affinity/taint conflicts or volume constraints. Compare the exact requested GPU resource key with node allocatable resources; a Ready CPU-only node cannot satisfy a GPU request.

Resolve the provider/device constraint. GKE standby buffers exclude GPU/TPU nodes; active buffers have separate provider requirements. The example buffer does not provision GPU capacity.

Pods run but are not ready

Inspect readiness and model-server logs:

kubectl get pod "$POD" -n "$NS" -o json | jq '{conditions: .status.conditions, containers: .status.containerStatuses}'
kubectl logs "$POD" -n "$NS" --all-containers=true --tail=100

Expect successful model loading and a passing readiness probe. Image-pull failures, download errors, OOMKilled or failed probes need different corrections; additional replicas do not fix them. Once the server is ready, repeat the tutorial’s model-list and completion requests.

Wait for model-ready evidence and test inference. Increase startup budgets only after identifying the delay.

Model metrics are absent or select the wrong model

Inspect exporter/collector and metric labels for the serving runtime.

Use the vLLM OTel tutorial and verify the query before changing scaling thresholds.

Latency worsens on scale-in or from zero

Separate queue time, time to first token, generation time and stream duration. Check termination/draining and timeout budgets.

Keep a measured warm floor when cold starts miss the objective; evaluate zero versus warm capacity.

Verify recovery

Repeat the same model request through the production route. Check successful responses, model-ready pods and time to first token; test cold starts separately from steady traffic.

For unresolved failures, Collect diagnostics for support with versions, namespace/target, timestamps, relevant conditions and a redacted configuration.