Azure 56: End-to-end deploy — from `git push v1.5.0` to canary promote in prod
easy⏱ 5 mincourseazure
The 12-step prod release
- git tag
v1.5.0 triggers the pipeline. 2. GitVersion confirms SemVer. 3. determine-version emits GIT_SEM_VER. 4. CI: docker build + push to uat & prod ACRs. 5. CD: terraform plan → exit 2 → publish plan artifact. 6. createprodcr generates the CR work item. 7. Pipeline pauses on QE + EM approval. 8. terraform apply runs. 9. AzureKeyVault@2 pulls secrets. 10. helm upgrade with --set secrets.*, isCanary=true. 11. Argo Rollouts at 10% → pause → analysis → promote → 50% → pause → promote → 100%. 12. scale-down delay finishes, old ReplicaSet GCd.
Each piece is owned
GitVersion + docker → CI team. terraform + CR → SRE. helm + Argo → platform. APIM + Front Door + service-bus topology → architecture. The pipeline-templates repo is the seam where all these owners agree on the interface (parameters in, stages out). One change to any piece cascades through 35+ services without coordination — exactly the property an enterprise stack needs.
Failure modes
GitVersion fails: tag is malformed → fix tag, retag, rerun. Docker push fails: ACR token expired → rotate in service connection. Terraform plan errors: state lock or syntax → unlock or fix HCL. Apply fails: typically RBAC; rerun after granting. helm fails: usually a missing Key Vault secret → add to mapping. Argo aborts: analysis failed → look at p99/error-rate dashboards, fix code, retag minor.
Build the orchestrator
Write a DeployOrchestrator class with one method per stage (gitVersion, buildAndPush, tfPlan, createCR, waitForApproval, tfApply, keyVaultPull, helmUpgrade, canaryStep, promote, finish). Each appends to an internal log with elapsed ms. Build a run({ env, service, version, simulateFailures }) method that walks them in order, stops at the first failure, and returns { outcome: 'shipped' | 'aborted', log, totalMs, finalWeight }. Log a happy-path prod release for cart-api v1.5.0 and a release where canary analysis fails at step 11.
Always link the run back to the tag
Argo Rollouts annotation rollout.argoproj.io/revision should match the docker tag, which should match the git tag, which should match the CR title. If you're ever debugging in prod and these don't all read 1.5.0, somebody is lying — usually the canary aborted halfway and someone restarted it without re-tagging.
Quiz: rolling back at step 11
Q: Analysis fails at 50% canary. Argo aborts. What now? A: The stable subset still has 100% of the old code; you have no user impact. Pipeline marks the run failed, leaves the CR open with the abort reason. On-call decides: roll forward (fix + retag) or roll back the Liquibase migration if step 8 changed schema. Prod is already at the safe state.