Azure 41: Self-Hosted ADO Build Agents on VMSS — pipeline-agents
easy⏱ 5 mincourseazure
Anatomy of pipeline-agents/tf/<env>/
Each env folder contains vmss.tf (the scale set), networking.tf (subnet + NSG that allows outbound to ACR/AKS/feeds), image.tf (data source pointing at the latest docker-jammy from platform-packer's RG), export-to-1password.tf (publishes the SSH key for break-glass), and per-env variables.tf (region, SKU, naming).
# tf/prod/vmss.tf (simplified)
resource "azurerm_linux_virtual_machine_scale_set" "lmr-build" {
name = "lmr-${var.NAMING_PREFIX}-vmss"
sku = var.IMAGE_SKU # Standard_D8ds_v5
instances = 0 # ADO scales 0..N
source_image_id = data.azurerm_image.image.id # docker-jammy
upgrade_mode = "Manual" # rolling updates done by us
overprovision = false
os_disk { storage_account_type = "StandardSSD_LRS" }
}
Custom image baked by Packer
docker-jammy is an Ubuntu 22.04 image with Docker, Terraform, Gradle 8, Node 22, .NET 8, kubectl, helm, az CLI, and the ADO agent bootstrap script pre-installed. It's built by the platform-packer repo into the platform-packer resource group. New tooling? Add it to the Packer template, version-bump the image, re-deploy the VMSS — your entire fleet upgrades on the next instance refresh.
ADO registers the pool, scales the fleet
In ADO project settings → Agent Pools, the pool driveway-pipeline-agents-prod points at the VMSS by resource ID. ADO scales the VMSS based on queued jobs: idle agents are deallocated, new jobs trigger a fresh instance from the source image. Reduces idle cost to near zero and gives every job a clean VM.
Build a VMSS spec validator
Write validateAgentPool(spec) that takes { name, env, sku, sourceImage, upgradeMode, location, tags } and returns { ok, errors }. Rules: name starts with lmr-, env in dev/uat/prod, prod SKU must be Standard_D8ds_v5 (smaller envs may use Standard_D4ds_v5), sourceImage must end with docker-jammy, upgradeMode must be Manual, location must be westus2, tags must include owner and costCenter. Log validation for two valid pools and two invalid ones.
Use Manual upgrade_mode in prod
With Automatic, a single Packer bump propagates to your fleet immediately — including in the middle of a deploy. With Manual, you trigger the image refresh from a maintenance window pipeline (pipeline-agents/refresh.yml) and watch metrics on the way up. Lythia learned the hard way: a bad Gradle minor version in the image once stalled every backend pipeline for 2 hours.
Quiz: where does the agent token come from?
Each VMSS instance boots, runs cloud-init, and registers with ADO. How does it authenticate without a baked-in PAT? Answer: the bootstrap script uses system-assigned managed identity to fetch a short-lived PAT from a Key Vault secret rotated nightly, or uses ADO's newer Microsoft Entra-authenticated agent protocol. No long-lived tokens on disk.