# OTel scaler configuration Configure `kedify-otel` triggers, metric queries and ingestion below. Use [OpenTelemetry metric scaling](https://docs.kedify.io/scalers/otel-scaler/) to understand the signal model and [choose an ingestion method](https://docs.kedify.io/how-to/otel-scaler-integrations/) before configuring the pipeline. The query syntax is a subset of PromQL. Collector/exporter endpoints and scaler query endpoints serve different purposes. ## Verify the contract Check a fresh source sample and the exact labels/units selected by the query before interpreting desired replicas. An OTLP receiver endpoint is different from the external-scaler endpoint even when one add-on provides both. Use the [general non-GPU tutorial](https://docs.kedify.io/how-to/scale-with-otel/) or [vLLM model-metric tutorial](https://docs.kedify.io/how-to/otel-scaler-general/), then [workload diagnostics](https://docs.kedify.io/troubleshooting/workload-scaling/) for missing/stale signals. ## Metric Ingestion In order to be able to scale workloads using custom metrics, these need to first be present in the scaler’s short-term in-memory db. Scaler exposes a regular gRPC OTLP receiver endpoint so there are several ways to set up your metric pipelines: - PUSH based (OTLP receiver) - PULL based (most of the time using the Prometheus [receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver)) This is not a complete list of all possible ways to gather the metrics, you can use all the available receivers in OTel ecosystem. ### Direct Metrics Push Kedify OTel scaler exposes OTLP receiver on port 4317 using gRPC protocol, so metrics can be pushed directly from user workloads. OpenTelementry instrumentation SDKs for various languages provide similar ways to configure the tool. One can configure the timeout, headers, SSL but most importantly the actual destination where the telemetry signals should be pushed. It is also possible to send metrics somewhere else than logs, traces and profiles. For metrics you can use the `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` environment variable. For more on this topic consult [this](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/) and [this](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/). Clear benefit of this approach is faster reaction time and no necessity for another OTel collector. On the other hand one can’t simply configure the fan out pattern where metrics will be sent to OTel scaler but also somewhere else for different purposes. Different languages and frameworks can have different ways to configure their observability stack. But environment variables should also work if they claim to be OTel compliant. ### Sidecar Pattern A lightweight sidecar container with OTel collector is deployed together with the workload. This collector scrapes the metrics exposed by the workload using a loopback interface (example: `http://localhost:8080/metrics`) and send them to the scaler. The sidecar collector can also do the filtering and send only the metric required for scaling. OTel Operator’s admission webhook can be used to inject such sidecars. This approach also requires the `cert-manager` to be present in the k8s cluster. Example `values.yaml` for deploying such setup: ```yaml otelOperator: enabled: true manager: env: [] admissionWebhooks: create: true otelOperatorCrs: - name: otel-sidecar-template enabled: true mode: sidecar namespace: default prometheusScrapeConfigs: - job_name: "otel-collector" scrape_interval: 5s static_configs: - targets: ["localhost:8000"] includeMetrics: - vllm:gpu_cache_usage_perc - vllm:num_requests_waiting ``` This also requires that pods that should receive the sidecar container have following annotation on them: `sidecar.opentelemetry.io/inject: "otel-sidecar-template"`. For more details please consult [docs](https://github.com/open-telemetry/opentelemetry-operator?tab=readme-ov-file#sidecar-injection) of OTel Operator. ### Static Targets List each scrape target as `host:port`. The default path is `/metrics`; use `metrics_path` in the scrape job for a different path, as described in the [Prometheus configuration reference](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#static_config). Example `values.yaml` for deploying such setup: ```yaml otelOperator: enabled: true otelOperatorCrs: - name: scrape-static-targets enabled: true prometheusScrapeConfigs: - job_name: "services" scrape_interval: 5s static_configs: - targets: ["keda-otel-scaler.keda.svc:8080"] - targets: ["kedify-predictor.keda.svc:8081"] - targets: ["my-workload.prod.svc:8080"] includeMetrics: - http_requests_total - keda_internal_metricsservice_grpc_client_msg_received_total - active_shopping_carts ``` More advanced [example](https://github.com/kedify/otel-add-on/blob/8cab1fe180a974436b9362cfa8694b4fce66a384/examples/vllm/dcgm/dcgm-values.yaml#L28) that scrapes the NVIDIA DCGM metrics. ### Kubernetes Service Discovery Prometheus OTel receiver supports full [configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration) as official Prometheus server. This allows to use their Kubernetes Service Discovery [feature](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) to dynamically specify the targets for scraping. Here is an example that will scrape any pods that have certain annotation on them: ```yaml ... prometheusScrapeConfigs: - job_name: 'k8s' scrape_interval: 5s kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_kedify_io_scrape] regex: "true" action: keep - source_labels: [__meta_kubernetes_pod_annotation_kedify_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_kedify_io_port] action: replace target_label: __address__ regex: (.+)(?::\d+);(\d+) replacement: $1:$2 - source_labels: [__meta_kubernetes_pod_annotation_kedify_io_scheme] action: replace target_label: __scheme__ regex: (.+) # all the 'implicit' labels starting with __ will be dropped, so we need to preserve the pod identity - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: pod_name ``` ([full config](https://github.com/kedify/otel-add-on/blob/main/examples/metric-pull/scaler-with-collector-pull-values.yaml)) Then all the pods with `kedify.io/scrape: "true"` annotation will be scraped for `"http//:8080/metrics"` endpoint. Which can be further configured using annotations: - `kedify.io/path` (default: `"/metrics"`) - `kedify.io/port` (default: `"8080"`) - `kedify.io/scheme` (default: `"http"`) ### Target Allocator Yet another approach to get the metrics in, is using the [Target Allocator](https://opentelemetry.io/docs/platforms/kubernetes/operator/target-allocator/) feature of OTel Operator. This way, one can describe the targets for metrics scraping using Prometheus Operator’s CRDs: - `PodMonitor` - `ServiceMonitor` This approach doesn’t require the Prometheus Operator nor Prometheus Server to be running in the cluster, however the CRDs are assumed to be present by OTel Operator. Full example: ```bash helm upgrade -i prometheus oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack --version 80.6.0 \ --set kubeStateMetrics.enabled=false \ --set nodeExporter.enabled=false \ --set grafana.enabled=false \ --set alertmanager.enabled=false kubectl scale statefulset/prometheus-prometheus-kube-prometheus-prometheus deploy/prometheus-kube-prometheus-operator --replicas=0 cat <