Azure 57: Datadog APM wired through the universal chart
easy⏱ 5 mincourseazure
The four pieces of Datadog wiring
(1) dd-java-agent.jar copied into the runtime stage of the Dockerfile so the JVM can attach it. (2) Conditional -javaagent:dd-java-agent.jar in ENTRYPOINT when env APM=true. (3) Rollout pod labels tags.datadoghq.com/env, tags.datadoghq.com/service, tags.datadoghq.com/version — Datadog's Universal Service Tagging. (4) Pipeline --set secrets.DATADOG_API_KEY=$(dd_api_key) from the datadog-agents-{env} Variable Group.
# Dockerfile runtime stage (eclipse-temurin-21)
COPY --from=builder /workspace/build/libs/dd-java-agent.jar /app/
ENV APM=true
ENTRYPOINT ["sh", "-c", "if [ \"$APM\" = \"true\" ]; then exec java -javaagent:/app/dd-java-agent.jar -jar /app/app.jar; else exec java -jar /app/app.jar; fi"]
Universal Service Tagging labels
The api universal chart's Rollout template includes three pod labels driven by chart values. They MUST match the conventions Datadog expects — typo tags.datadoghq.com/env to env: prod and your dashboards split into two services. The chart pulls them from envName, applicationName helper, and the image tag — so the labels are always coherent with the deploy.
metadata:
labels:
app: {{ include "applicationName" . }}
tags.datadoghq.com/env: {{ .Values.envName }}
tags.datadoghq.com/service: {{ include "applicationName" . }}
tags.datadoghq.com/version: {{ .Values.deployment.image.tag }}
API key flow
The shared Variable Group datadog-agents-{env} holds one secret: dd_api_key. Every service's pipeline lists this group in variableGroups: and adds --set secrets.DATADOG_API_KEY=$(dd_api_key) to its helm overrides. The chart writes it into a K8s Secret; the Datadog agent DaemonSet on each node reads it at startup. One key, one rotation, every pod tagged.
Validate the wiring
Write validateDatadogWiring({ dockerfile, rolloutLabels, pipelineVarGroups, env }) that returns { ok, missing: string[] }. Check for: dd-java-agent.jar reference in dockerfile, -javaagent: in entrypoint, all three tags.datadoghq.com/{env,service,version} labels in rolloutLabels, and datadog-agents-<env> in pipelineVarGroups. Log a fully-wired service and a misconfigured one.
Datadog Agent must run as DaemonSet on each node
Lythia installs the Datadog agent via the official Helm chart as a DaemonSet, NOT as a sidecar per pod. The agent listens on a host UNIX socket; the universal chart's APM init does DD_AGENT_HOST=$(NODE_IP) so each pod talks to the local agent. Avoid sidecar mode — it doubles your pod count and the resource overhead is real on large fleets.
Quiz: same service tagged twice in Datadog
Q: After a release, Datadog shows two cart-api entries — one version: 1.4.0, one version: 1.5.0. Is that a bug? A: No, it's correct during a canary. Argo Rollouts is running both versions simultaneously. The dashboard's Service Map will show the split. Once promote finishes and the old ReplicaSet GCs, the 1.4.0 entry stops emitting and Datadog rolls it off the dashboard.