Multi-cluster setup with GitOps

This guide shows a GitOps-friendly way to add member clusters for Kedify multicluster in 2 steps:

1. Apply static resources to each member cluster.
2. Store auth values in a Git-managed Secret in the KEDA Argo app.

Prerequisites

• Access to the member cluster.
• Access to the KEDA cluster.
• kubectl, base64.
• KEDA namespace (defaults to keda).

1. Apply static resources on the member cluster

Apply these manifests in the member cluster (for example from an Argo app).

apiVersion: v1
kind: Namespace
metadata:
  name: keda
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kedify-agent
  namespace: keda
---
apiVersion: v1
kind: Secret
metadata:
  name: kedify-agent-token
  namespace: keda
  annotations:
    kubernetes.io/service-account.name: kedify-agent
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kedify-agent
rules:
- apiGroups: ["*"]
  resources: ["*/scale"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kedify-agent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kedify-agent
subjects:
- kind: ServiceAccount
  name: kedify-agent
  namespace: keda

Once the kedify-agent-token Secret is applied to the member cluster, the serviceaccount controller populates it with a generated auth token and CA bundle. In the next step, copy these values to kedify-agent-multicluster-kubeconfigs in the KEDA cluster.

2. Retrieve dynamic auth values from the member cluster

All member cluster kubeconfigs are stored in one secret in the KEDA cluster: kedify-agent-multicluster-kubeconfigs. This way, Kedify Agent does not need broad access to all Secrets in the namespace.

Describe the remote cluster:

MEMBER_NAME="member-a"
MEMBER_CONTEXT="member-a-context"
NAMESPACE="keda"

Read token, CA and server address:

TOKEN=$(kubectl --context "$MEMBER_CONTEXT" -n "$NAMESPACE" get secret kedify-agent-token -o jsonpath='{.data.token}' | base64 -d)
CA_DATA=$(kubectl --context "$MEMBER_CONTEXT" -n "$NAMESPACE" get secret kedify-agent-token -o jsonpath='{.data.ca\.crt}')
SERVER=$(kubectl config view --raw --minify --context="$MEMBER_CONTEXT" -o jsonpath='{.clusters[0].cluster.server}')

Generate and encode kubeconfig for Git:

KCFG_B64=$(echo -n "apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: ${CA_DATA}
    server: ${SERVER}
  name: ${MEMBER_NAME}-cluster
contexts:
- context:
    cluster: ${MEMBER_NAME}-cluster
    user: kedify-agent
  name: kedify-agent@${MEMBER_NAME}
current-context: kedify-agent@${MEMBER_NAME}
users:
- name: kedify-agent
  user:
    token: ${TOKEN}
" | base64 | tr -d '\n')

Put the value into a Git-managed Secret manifest used by your KEDA Argo CD app:

apiVersion: v1
kind: Secret
metadata:
  name: kedify-agent-multicluster-kubeconfigs
  namespace: keda
type: Opaque
data:
  member-a-cluster.kubeconfig: <BASE64_OF_MEMBER_A_KUBECONFIG>
  member-b-cluster.kubeconfig: <BASE64_OF_MEMBER_B_KUBECONFIG>

Replace each <BASE64_OF_...> placeholder with the output from KCFG_B64 for that member cluster.

3. Verify

Check that the member key exists:

kubectl kedify mc list-members