Azure 43: AKS Multi-Cluster Topology — apps + elasticsearch
easy⏱ 5 mincourseazure
Why split apps from search
Elasticsearch workloads have very different shape: huge memory (i-series VMs), local SSD, slow rolling restarts. Customer APIs need fast autoscale, low memory, and frequent deploys. Mixing them in one cluster means an ES heap spike GC-pauses a checkout API. Two clusters = blast-radius isolation, independent upgrade timing, separate IAM.
Service connections by purpose
Pipelines look up the right kubeconfig string by environment and by cluster purpose. pipeline-templates/variables/global.yaml declares them: k8sAppsProdSC, k8sAppsUatSC, k8sEsProdSC etc. A backend microservice helm deploy uses k8sAppsProdSC. An ES index migration job uses k8sEsProdSC. Never let an app pipeline touch the ES cluster, even with dry-run.
# pipeline-templates/variables/global.yaml
k8sAppsDevSC: AKS-apps-k8s-dev
k8sAppsProdSC: AKS-apps-k8s-prod
k8sEsDevSC: AKS-es-k8s-dev
k8sEsProdSC: AKS-es-k8s-prd2 # historical typo, kept stable
# helm-deploy-v2.yaml picks at runtime:
azureSubscription: $(k8sAppsProdSC)
Region pinning
Both clusters live in westus2. The VNet exists once, and each cluster is attached via its own subnet (/22 per node pool, room to grow to ~1000 pods). Outbound to ACR + Service Bus + PG Flex flows through service endpoints; inbound traffic only reaches AKS through APIM → Front Door, so node IPs are never published.
Build the cluster router
Write routeCluster(env, workloadKind) where workloadKind is 'app' | 'search'. Return { sc: '...', purpose, env, rationale }. Validate env in dev/test/uat/prod. Log routing for ('prod','app'), ('uat','search'), ('dev','app'). Then attempt ('staging','app') and capture the error.
Different upgrade cadences
Lythia upgrades AKS-apps on a weekly cycle (off-peak) and AKS-es on a monthly cycle (after a snapshot). Misaligning them isolates upgrade risk and means you can roll back one without dragging the other.
Quiz: a job pushed an image with kubectl set image instead of helm
Q: Why did monitoring stop seeing it on the dashboard? A: Datadog tags come from the Helm rollout's pod labels, not from kubectl. The image is running, but tags.datadoghq.com/version points to the stale chart value. Always deploy via the universal chart; never let kubectl set image near a cluster Argo Rollouts owns.