Multi-cluster setup with GitOps

This guide shows two GitOps-friendly ways to add member clusters for Kedify multicluster. Both rely on the same member-cluster preparation, only the KEDA-cluster-side wiring differs.

• Per-cluster Secret (recommended): one labeled Secret per member, picked up by Kedify Agent’s kubeconfig provider. Members can be added or removed without touching a bundled Secret or restarting the agent.
• Bundled Secret: a single kedify-agent-multicluster-kubeconfigs Secret carrying all member kubeconfigs under separate data keys, read from a mounted volume by Kedify Agent’s file provider.

Both paths can be used at the same time on the same agent.

About the two providers

Each path is backed by a different provider from the upstream sigs.k8s.io/multicluster-runtime library that Kedify Agent embeds. The provider names reflect where the agent discovers cluster credentials from, not what the credentials look like (both are kubeconfigs).

• Per-cluster Secret is handled by the kubeconfig provider, which watches labeled Secrets in the agent’s install namespace and treats each Secret’s data.kubeconfig payload as a member.
• Bundled Secret is handled by the file provider, which watches a directory of *.kubeconfig files on disk. The bundled kedify-agent-multicluster-kubeconfigs Secret is mounted as that directory, so each data key shows up as a separate file the provider picks up.

The provider name surfaces as the provider field on each KedifyConfiguration.status.multiClusterStatus.clusters[] entry and on each DSO/DSJ MemberClusterStatus. It also appears as the prefix in canonical cluster IDs (kubeconfig#…, file#…).

Prerequisites

• Access to the member cluster.
• Access to the KEDA cluster.
• kubectl, base64, jq.
• Kedify Agent installed with agent.features.distributedScaledObjectsEnabled or agent.features.distributedScaledJobsEnabled set to true.

1. Apply static resources on the member cluster

Apply these manifests in the member cluster (for example from an Argo app). The same set is used regardless of which KEDA-cluster path you choose below.

apiVersion: v1
kind: Namespace
metadata:
  name: keda
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kedify-agent
  namespace: keda
---
apiVersion: v1
kind: Secret
metadata:
  name: kedify-agent-token
  namespace: keda
  annotations:
    kubernetes.io/service-account.name: kedify-agent
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kedify-agent
rules:
- apiGroups: ["*"]
  resources: ["*/scale"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kedify-agent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kedify-agent
subjects:
- kind: ServiceAccount
  name: kedify-agent
  namespace: keda

Once the kedify-agent-token Secret is applied, the serviceaccount controller populates it with a generated auth token and CA bundle.

2. Retrieve dynamic auth values from the member cluster

Describe the remote cluster:

MEMBER_NAME="member-a"
MEMBER_CONTEXT="member-a-context"
NAMESPACE="keda"

Read token, CA and server address:

TOKEN=$(kubectl --context "$MEMBER_CONTEXT" -n "$NAMESPACE" get secret kedify-agent-token -o jsonpath='{.data.token}' | base64 -d)
CA_DATA=$(kubectl --context "$MEMBER_CONTEXT" -n "$NAMESPACE" get secret kedify-agent-token -o jsonpath='{.data.ca\.crt}')
SERVER=$(kubectl config view --raw --minify --context="$MEMBER_CONTEXT" -o jsonpath='{.clusters[0].cluster.server}')

Generate and encode kubeconfig for Git:

KCFG_B64=$(echo -n "apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: ${CA_DATA}
    server: ${SERVER}
  name: ${MEMBER_NAME}-cluster
contexts:
- context:
    cluster: ${MEMBER_NAME}-cluster
    user: kedify-agent
  name: kedify-agent@${MEMBER_NAME}
current-context: kedify-agent@${MEMBER_NAME}
users:
- name: kedify-agent
  user:
    token: ${TOKEN}
" | base64 | tr -d '\n')

3. Register the member in the KEDA cluster

Per-cluster Secret

Create one labeled Secret per member in the KEDA cluster in the same namespace where Kedify Agent is deployed. Each Secret carries one member’s kubeconfig under the kubeconfig data key, and the Secret’s metadata.name becomes the cluster’s alias used in spec.memberClusters[].name.

apiVersion: v1
kind: Secret
metadata:
  name: member-a                                          # this becomes the cluster alias
  namespace: keda
  labels:
    sigs.k8s.io/multicluster-runtime-kubeconfig: "true"   # selector the agent watches
type: Opaque
data:
  kubeconfig: <BASE64_OF_MEMBER_KUBECONFIG>

Apply one Secret per member you want to register. The agent’s kubeconfig provider picks the Secret up immediately and engages the cluster; no agent restart is needed and no other Secret needs editing when a member is added or removed.

Bundled Secret

Members registered via this path share a single Secret in the KEDA cluster: kedify-agent-multicluster-kubeconfigs. The agent’s file provider reads them from a mounted volume. Members registered via the per-cluster Secret path coexist independently and are not affected by edits to this Secret.

apiVersion: v1
kind: Secret
metadata:
  name: kedify-agent-multicluster-kubeconfigs
  namespace: keda
type: Opaque
data:
  member-a-cluster.kubeconfig: <BASE64_OF_MEMBER_A_KUBECONFIG>
  member-b-cluster.kubeconfig: <BASE64_OF_MEMBER_B_KUBECONFIG>

Replace each <BASE64_OF_...> placeholder with the output from KCFG_B64 for that member cluster. Adding or removing a member requires editing this single Secret; the agent picks up the change via the mounted volume’s filesystem notification.

4. Verify

kubectl kedify mc list-members

Or read the status directly:

kubectl -n keda get kedifyconfiguration kedify \
  -o json | jq '.status.multiClusterStatus.clusters'

Each entry carries a provider field showing which provider registered the cluster (file or kubeconfig).

Reference forms in spec.memberClusters[].name

DistributedScaledObject and DistributedScaledJob accept three equivalent forms when referencing a cluster:

Form	Example
Alias	member-a
Canonical	kubeconfig#member-a or file#/etc/mc/kubeconfigs/member-a-cluster.kubeconfig+kedify-agent@member-a
Provider-local	identical to the alias for the kubeconfig provider; for the file provider: /etc/mc/kubeconfigs/member-a-cluster.kubeconfig+kedify-agent@member-a

The alias form is the everyday one. Canonical form is the disambiguating form when you have a cross-provider name collision (see below).

Cross-provider name collisions

If both the bundled file Secret and a per-cluster kubeconfig Secret register the same name, the agent picks the kubeconfig entry. The file cluster stays in the inventory and remains reachable through its canonical ID. A Warning event with reason MultiClusterNameCollision is recorded on KedifyConfiguration, and the file entry’s ClusterStatus.Info notes the masked alias.

To target the file cluster explicitly during a collision, use its canonical form in spec.memberClusters[].name (for example file#/etc/mc/kubeconfigs/member-a-cluster.kubeconfig+kedify-agent@member-a).

The simplest remediation is to pick different names: rename the per-cluster Secret, or remove the conflicting entry from the bundled Secret. Once one side is gone, the next status tick clears the collision and the surviving cluster reclaims the plain alias.