Multi-Cluster Scaling

Kedify supports scaling workloads across a fleet of Kubernetes clusters. This is achieved through a new DistributedScaledObject custom resource that extends the standard ScaledObject with multi-cluster capabilities.

Architecture

There are two main types of clusters involved in multi-cluster scaling:

1. KEDA Cluster: This cluster runs the Kedify stack and manages the scaling logic. It monitors the metrics and decides when to scale workloads up or down.
2. Member Clusters: These clusters host the actual workloads that need to be scaled. They expose their kube-apiserver to the KEDA cluster for management.

The member clusters don’t need to run KEDA themselves, as scaling decisions for DistributedScaledObject are made by the KEDA cluster. This allows for a smaller footprint on member clusters and enables edge scenarios where resources are limited.

Connecting Member Clusters

In order to connect a member cluster to the KEDA cluster, you need to make the kube-apiserver of the member cluster accessible from the KEDA cluster. This can be done using various methods such as VPN, VPC peering, or exposing the API server via a load balancer with proper security measures.

With the connectivity established, you can use Kedify’s kubectl plugin to register member clusters to the KEDA cluster:

kubectl kedify mc setup-member <name> --keda-kubeconfig <path> --member-kubeconfig <path>

This command will use the provided kubeconfig files to set up the necessary access and permissions for the KEDA cluster to manage the member cluster. The member-kubeconfig should have sufficient permissions to create RBAC, ServiceAccount and keda namespace in the member cluster, these resources will be created with minimal privileges required for Kedify multi-cluster to operate. The keda-kubeconfig should have permissions to patch Secret named kedify-agent-multicluster-kubeconfigs in keda namespace in the KEDA cluster. For connecting multiple member clusters, you can repeat the above command with different names and kubeconfig files for each member cluster.

In case you would like the KEDA cluster to connect to the member cluster using a different address than the one specified in the member-kubeconfig, you can provide --member-api-url <url> flag to override the API server URL.

You can also list and remove registered member clusters using the following commands:

kubectl kedify mc list-members
kubectl kedify mc delete-member <name>

DistributedScaledObject

A ScaledObject is a KEDA resource that defines how to scale a specific workload based on certain metrics. The DistributedScaledObject extends this concept to support scaling across multiple clusters. It includes all the fields of a standard ScaledObject, along with additional fields to specify the member clusters and their configurations.

Here is an example of a DistributedScaledObject:

apiVersion: keda.kedify.io/v1alpha1
kind: DistributedScaledObject
metadata:
  name: nginx
spec:
  memberClusters:             # optional list of member clusters to use, if omitted all registered member clusters will be used
  - name: member-cluster-1
    weight: 4                 # weight determines the proportion of replicas to be allocated to this cluster
  - name: member-cluster-2
    weight: 6
  rebalancingPolicy:          # optional parameters for rebalancing replicas across member clusters in case of outage or issues
    gracePeriod: 1m           # when a member cluster becomes unreachable, wait for this duration before rebalancing replicas to other clusters
  scaledObjectSpec:           # standard ScaledObject spec
    scaleTargetRef:
      kind: Deployment
      name: nginx
    minReplicaCount: 1
    maxReplicaCount: 10
    triggers:
    - type: kubernetes-resource
      metadata:
        resourceKind: ConfigMap
        resourceName: mock-metric
        key: metric-value
        targetValue: "5"

In this example, the DistributedScaledObject named nginx is configured to scale a Deployment named nginx across two member clusters. The memberClusters field whitelists the member clusters to be used along with their respective weights, which determine how many replicas should be allocated to each cluster. This section is optional; if omitted, all registered member clusters will be used with equal weights.

The workloads of type Deployment are expected to be present in relevant member clusters in a matching namespace as the DistributedScaledObject.

The rebalancingPolicy field allows you to specify how to handle situations where a member cluster becomes unreachable. In this case, after the specified gracePeriod, the replicas that were allocated to the unreachable cluster will be redistributed among the remaining healthy clusters. Once the unreachable cluster becomes healthy again, the replicas will be rebalanced back according to the defined weights.

Status of the DistributedScaledObject provides insights into the scaling state across member clusters:

status:
  memberClusterStatuses:
    member-cluster-1:
      currentReplicas: 2
      description: Cluster is healthy
      desiredReplicas: 2
      id: /etc/mc/kubeconfigs/member-cluster-1.kubeconfig+kedify-agent@member-cluster-1
      lastStatusChangeTime: "2025-11-05T16:46:39Z"
      state: Ready
    member-cluster-2:
      currentReplicas: 3
      description: Cluster is healthy
      desiredReplicas: 3
      id: /etc/mc/kubeconfigs/member-cluster-2.kubeconfig+kedify-agent@member-cluster-2
      lastStatusChangeTime: "2025-11-05T15:45:44Z"
      state: Ready
  membersHealthyCount: 2
  membersTotalCount: 2
  selector: kedify-agent-distributedscaledobject=nginx
  totalCurrentReplicas: 5

For a walkthrough example on how to set up and use multi-cluster scaling with Kedify, refer to the examples repository.