# Send metric history to the predictor with OTLP

There are currently three ways to get metrics for Predictor:

- from static CSV file

- dynamically from existing KEDA trigger (either defined on ScaledObject or ScaledJob)

- **using OTLP receiver**

This tutorial will focus on the last method - OTLP gRPC-based receiver.

## Prerequisite

Make sure the Kedify Agent and Kedify Predictor are both correctly installed and running. To do that you may want to consult [installation](https://docs.kedify.io/how-to/predictive-scaler-installation/) docs.

OTLP receiver is enabled by default and is agnostic about the metric storage Predictor uses, however, if you want to change either port, kubernetes service or add TLS settings, this can be done using the custom Helm chart values for the Kedify Predictor.

Example:

```yaml
kedaPredictionController:
  otlpReceiver:
    enabled: true
    port: 4317
    # nodePort: 31112
    tls:
      # (optional) path to CA certificate. When provided, the client certificate will be verified using this CA where "client" ~ another OTLP exporter.
      caFile: ""
      # (optional) path to TLS certificate that will be used for OTLP receiver
      certFile: ""
      # (optional) path to TLS key that will be used for OTLP receiver
      keyFile: ""
      # (optional) specifies the duration after which the certificates will be reloaded.
      # This is useful when using the CertManager for rotating the certs mounted as Secrets.
      reloadInterval: "5m"
```

## Metric Pipeline Setup

We will deploy a custom Prometheus exporter that periodically fetches the weather information and exposes them as metrics. Each ten minutes it makes a call to openweathermap.org API about the weather in San Franciso.

### Metric Source

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: weather-exporter
spec:
  selector:
    matchLabels:
      app: weather-exporter
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: weather-exporter
    spec:
      containers:
        - name: weather-exporter
          image: ghcr.io/kedify/weather-prometheus-exporters
          args:
            - -config=/cfg/config.json
            - -addr=:8080
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: weather-exporter-config
              mountPath: /cfg
          env:
            - name: OPEN_WEATHER_APP_ID
              value: 4524f75e02d8b944ee74bb0d3ddc6efc
      volumes:
        - name: weather-exporter-config
          configMap:
            name: weather-exporter-config
            defaultMode: 420
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: weather-exporter-config
  namespace: default
data:
  # SanFrancisco coords
  config.json: |
    {
      "OpenWeather": {
        "CurrentWeatherData": {
          "Enabled": true,
          "Coords": [{ "Lat": 37.7739, "Lon": -122.4312 }],
          "Interval": "600s"
        }
      }
    }
```

```bash
# create also a service
kubectl expose deploy/weather-exporter --port 8080 --name weather-exporter
```

We can check the exposed metrics by:

```bash
kubectl port-forward svc/weather-exporter 8080
curl localhost:8080

# among others, there should be a metric called 'open_weather_clouds_all' that we are going to use
# this metric denotes the current percentage rate of 'cloudiness' in SF (0 -> clear sky)
# open_weather_clouds_all{id="5391959",name="San Francisco"} 40
```

And finally, deploy the OTel collector that will be scraping it and forwarding it to Predictor. With the following configuration, it will scrape the metrics each minute.

```bash
cat <<VALS | helm upgrade -i otelcol oci://ghcr.io/open-telemetry/opentelemetry-helm-charts/opentelemetry-collector --version=0.143.0 -f -
  image:
    repository: otel/opentelemetry-collector-contrib
  mode: deployment
  alternateConfig:
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'weather'
              scrape_interval: 60s
              static_configs:
                - targets: ['weather-exporter:8080']
    exporters:
      otlp/kedify-predictor:
        endpoint: kedify-predictor.keda.svc:4317
        compression: "none"
        tls:
          insecure: true
      debug:
        verbosity: detailed
    processors:
      filter/ottl:
        error_mode: ignore
        metrics:
          metric:
            - name != "open_weather_clouds_all"
    extensions:
      health_check:
        endpoint: \${env:MY_POD_IP}:13133
    service:
      extensions: [health_check]
      pipelines:
        metrics:
          receivers:
            - prometheus
          processors:
            - filter/ottl
          exporters: [debug, otlp/kedify-predictor]
VALS
```

Then in OTel collector logs you should be able to see:

```bash
2026-01-20T15:36:02.018Z  info  ResourceMetrics #0
Resource SchemaURL:
Resource attributes:
     -> service.name: Str(weather)
     -> server.address: Str(weather-exporter)
     -> service.instance.id: Str(weather-exporter:8080)
     -> server.port: Str(8080)
     -> url.scheme: Str(http)
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver 0.143.0
Metric #0
Descriptor:
     -> Name: open_weather_clouds_all
     -> Description:
     -> Unit:
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> id: Str(5391959)
     -> name: Str(San Francisco)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2026-01-20 15:36:02.002 +0000 UTC
Value: 75.000000
  {"resource": {"service.instance.id": "7301fd7f-b9e9-4703-b5c3-761e08c820ba", "service.name": "otelcol-contrib", "service.version": "0.143.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}
```

and for each “tick” also a corresponding log entry in `kedify-predictor` (container `prediction-controller`):

```bash
Data point count: 1
-  name: open_weather_clouds_all
   type: Gauge
     - time: 2026-01-20 15:36:02.002 +0000 UTC
       tags: map[id:5391959 name:San Francisco]
       value: 75
```

This means the metric is flowing from the Prometheus exporter app, via OTel collector that periodically scrapes it, to Kedify Predictor OTLP receiver. This, however, does not mean that Predictor stores it in its metric store. To do that, one has to first create the MetricPredictor custom k8s resource.

To do that:

```bash
cat <<MP | kubectl apply -f -
kind: MetricPredictor
apiVersion: keda.kedify.io/v1alpha1
metadata:
  name: cloudiness
spec:
  source:
    otel:
      metricName: open_weather_clouds_all
      requiredLabels:
        name: "San Francisco"
  model:
    type: Prophet
    name: cloudiness
    defaultHorizon: 1h
    retrainInterval: 2d
MP
```

## Predict

By creating the MetricPredictor resource, we effectively did a couple of important things:

- enabled the storage of incoming metric points for OTLP

- created the association that all incoming metric points for metric of type Gauge with name ‘`open_weather_clouds_all`’ will be used for model called ‘`cloudiness`’

- we restricted the metric to only those that have tag or label `name=San Francisco` on them

- and finally enabled the periodic retrain for the model

To check the status of the model:

```bash
λ k get mp -owide
NAME         LASTRETRAIN   MODELNAME    READY   SOURCE   SOURCEDETAIL              ACTIVESOURCES   AGE     STATE            RETRAININTERVAL
cloudiness   2m29s         cloudiness   True    OTel     open_weather_clouds_all   1               2m52s   SubscribedOTLP   48h0m0s
```

Once the model is first retrained and it’s MAPE (/performance) is not terrible, one can start using it. Using the trained model for predictions is the same as for the use-case with KEDA metric source.

This means, one can reference the model from kedify-predictive trigger the similar way as in [here](https://docs.kedify.io/how-to/predictive-scaler-example/#5-start-using-the-model-for-predictions).

```yaml
# part of Scaled{Object,Job}
triggers:
---
- type: kedify-predictive
  name: cloudiness
  metadata:
    modelName: cloudiness
```

This way we can create an autoscaling rule based on a real historical data. In this particular case it’s the cloudiness in San Francisco. Now, if the cloudiness in San Francisco exhibits a seasonal effects that can be predicted to some degree, Prophet will find it and suggest the future values. Predicting the weather is nearly impossible, but there will be probably more clouds during certain seasons or time of a day. If we own a solar power plant, we can scale our infrastructure upfront. Or mix the real measured data with the predicted data using KEDA scaling modifiers. Sky/clouds is the limit.

## Prometheus-write as Possible Metric Source

We have shown that we can use an arbitrary existing OTel receiver and forward the metrics to Kedify Predictor. The previous example was using the Prometheus receiver that can periodically scrape the metric endpoint using HTTP client that exposes the metrics in plain text.

If we need a metric coming from [prometheus-write](https://prometheus.io/docs/specs/prw/remote_write_spec/) protocol because our monitoring solution supports this. The example would be the same up to the OTel collector’s configuration.

```yaml
receivers:
  prometheusremotewrite:
    endpoint: 0.0.0.0:9090
exporters: ...
processors: ...
extensions: ...
service: ...
```

Then if the tool that is supposed to feed the metrics is also configured to send the metrics to OTel collector k8s service and port `9090`, it will be relayed to Kedify Predictor.

For all the available receivers please consult the upstream [docs](https://opentelemetry.io/docs/collector/components/receiver/).

## Continue with this topic

**Reference:** [OTel scaler configuration](https://docs.kedify.io/reference/otel-scaler/) · [Predictive scaler trigger configuration](https://docs.kedify.io/reference/predictive-scaler/) · [MetricPredictor API reference](https://docs.kedify.io/reference/kubernetes-api/metricpredictor/).

**Diagnose:** [Forecasts are missing or inaccurate](https://docs.kedify.io/troubleshooting/predictive-scaling/).

---
Canonical: https://docs.kedify.io/how-to/predictive-scaler-otlp/
Source: src/content/docs/how-to/predictive-scaler-otlp.md
Documentation index: https://docs.kedify.io/llms.txt