# Distribute jobs across clusters

This guide shows how to use `DistributedScaledJob` (DSJ) to process queue-based workloads across multiple member clusters.

[![The central DistributedScaledJob controller uses KEDA metrics to create Jobs across connected member clusters.](https://docs.kedify.io/assets/images/docs/distributed_scaled_job.svg)](https://docs.kedify.io/assets/images/docs/distributed_scaled_job.svg)

Scroll to exploreDiagram description

A central KEDA installation evaluates demand and uses DistributedScaledJob to update member Kubernetes APIs. Member clusters do not require another KEDA installation.

## Prerequisites

- Kedify Agent is installed in your KEDA cluster.

- Member clusters are registered through one of the [distributed registration providers](https://docs.kedify.io/how-to/multicluster-gitops/).

- `kubectl` installed and access to: 

  - KEDA cluster context

  - each member cluster context

## 1. Enable DSJ and raw metrics

`DistributedScaledJob` requires:

- `DSJ_ENABLED="true"` in `kedify-agent`

- KEDA raw metrics gRPC protocol enabled

For an Agent chart with bundled KEDA, select distributed mode in your maintained [Helm/GitOps values](https://docs.kedify.io/installation/helm/). The chart enables both requirements:

```yaml
global:
  features:
    multicluster:
      enabled: true
      type: distributed
```

If KEDA is installed separately, enable its raw metrics gRPC explicitly (values for the standalone KEDA chart):

```yaml
env:
  - name: RAW_METRICS_GRPC_PROTOCOL
    value: enabled
```

[Dedicated mode](https://docs.kedify.io/features/multi-cluster-scaling/#dedicated-mode) instead reconciles ordinary ScaledJobs in each member and does not use the Agent DSJ controller.

## 2. Prepare ConfigMap trigger

Create this ConfigMap in the KEDA cluster (`default` namespace in this example):

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: dsj-mock-metric
  namespace: default
data:
  metric-value: "0"
```

```bash
kubectl --context keda-cluster -n default apply -f dsj-mock-metric.yaml
```

## 3. Create DistributedScaledJob

Apply this `DistributedScaledJob` in the KEDA cluster:

```yaml
apiVersion: keda.kedify.io/v1alpha1
kind: DistributedScaledJob
metadata:
  name: dsj-processor
  namespace: default
spec:
  memberClusters:
    - name: member-1
      weight: 2
    - name: member-2
      weight: 3
  clusterScheduling:
    strategy: weightedRoundRobin
    failoverPolicy:
      gracePeriod: 1m
      hardTaintDuration: 5m
      softTaintDuration: 3m
  scaledJobSpec:
    pollingInterval: 30
    maxReplicaCount: 20
    successfulJobsHistoryLimit: 2
    failedJobsHistoryLimit: 2
    scalingStrategy:
      strategy: pendingAware
    jobTargetRef:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: processor
              image: busybox:1.36
              command:
                - /bin/sh
                - -c
                - |
                  echo "processing message";
                  sleep 10
    triggers:
      - type: kubernetes-resource
        name: cfg
        metadata:
          resourceKind: ConfigMap
          resourceName: dsj-mock-metric
          key: metric-value
          targetValue: "5"
```

```bash
kubectl --context keda-cluster -n default apply -f distributedscaledjob.yaml
```

### Use cluster-specific environment values

Jobs may need cluster-specific information, such as the cluster identity, region, or a local service endpoint. Store these values in a ConfigMap on each execution cluster, using the same name and keys but different data.

For example, to pass the execution cluster’s ID to a worker, create the following ConfigMap in the worker namespace on each cluster:

```bash
kubectl --context member-1 -n default create configmap worker-cluster-identity \
  --from-literal=clusterId=member-1
kubectl --context member-2 -n default create configmap worker-cluster-identity \
  --from-literal=clusterId=member-2
```

Add this environment entry to the worker container under `spec.scaledJobSpec.jobTargetRef.template.spec.containers[].env`, replacing any literal `K8S_CLUSTERID` entry:

```yaml
env:
  - name: K8S_CLUSTERID
    valueFrom:
      configMapKeyRef:
        name: worker-cluster-identity
        key: clusterId
```

Kubernetes resolves the reference in the cluster and namespace where the Pod runs, so the DSJ template can stay identical across clusters. Keep this ConfigMap separate from shared worker configuration that should have identical contents everywhere.

Create the ConfigMap and key before scheduling workers; a missing ConfigMap or key prevents the container from starting. Changes to the ConfigMap do not update environment values in existing Pods. See [ConfigMaps in Kubernetes](https://kubernetes.io/docs/concepts/configuration/configmap/#using-configmaps-as-environment-variables).

## 4. Trigger scaling

Increase the ConfigMap value above target (`5`):

```bash
kubectl --context keda-cluster -n default patch configmap dsj-mock-metric \
  --type merge \
  -p '{"data":{"metric-value":"20"}}'
```

Set it back below target:

```bash
kubectl --context keda-cluster -n default patch configmap dsj-mock-metric \
  --type merge \
  -p '{"data":{"metric-value":"0"}}'
```

When the value is above target, DSJ creates Jobs and distributes them across member clusters according to weights.

## 5. Verify status and distribution

Check DSJ status in the KEDA cluster:

```bash
kubectl --context keda-cluster -n default get distributedscaledjob dsj-processor -o yaml
```

Check created Jobs in each member cluster:

```bash
kubectl --context member-1 -n default get jobs
kubectl --context member-2 -n default get jobs
```

You should observe approximately weighted distribution (`member-1:2`, `member-2:3`) over time.

## Troubleshooting

- No Jobs are created: 

  - Verify `DSJ_ENABLED="true"` on `kedify-agent`.

  - Verify KEDA has `RAW_METRICS_GRPC_PROTOCOL=enabled`.

  - Verify `memberClusters[].name` matches registered member names exactly.

- Trigger does not fire: 

  - Verify ConfigMap exists in the same namespace as DSJ.

  - Verify `metric-value` is numeric and above `targetValue`.

- Jobs stuck in Pending: 

  - Check member cluster resources and scheduling constraints.

  - Review DSJ status and `clusterScheduling.failoverPolicy` behavior.

## Continue with this topic

**Configure and verify:** [Register member clusters with GitOps](https://docs.kedify.io/how-to/multicluster-gitops/).

**Reference:** [DistributedScaledJob API reference](https://docs.kedify.io/reference/kubernetes-api/distributedscaledjob/).

**Diagnose:** [Diagnose multi-cluster and tenant controllers](https://docs.kedify.io/troubleshooting/platform-scaling/).

**Related capabilities:** [Distributed multi-cluster scaling](https://docs.kedify.io/features/multi-cluster-scaling/).

---
Canonical: https://docs.kedify.io/how-to/multicluster-dsj/
Source: src/content/docs/how-to/multicluster-dsj.md
Documentation index: https://docs.kedify.io/llms.txt