Datadog Custom Metrics via Micrometer

Micrometer is to metrics what SLF4J is to logging: a vendor-neutral facade that your application code depends on, so that swapping the vendor underneath it never means touching business logic. You instrument `VehicleValuationService` and its provider clients once, against Micrometer's `Counter`, `Timer`, and `Gauge` types, and whichever `MeterRegistry` implementation is on the classpath decides where those numbers actually go. Point the same instrumented code at Prometheus, CloudWatch, or Datadog by swapping a dependency and a handful of properties — none of the calls to `Counter.builder(...)` scattered through the codebase change.

For this service, that registry is Datadog's: adding `micrometer-registry-datadog` to the classpath and setting the API key and step interval gives Spring Boot's autoconfiguration everything it needs to wire up a `DatadogMeterRegistry` automatically. It is worth knowing this registry pushes — on a fixed interval, it calls Datadog's HTTP API directly — which is a different model from Prometheus's pull-based scraping, and it is why the `step` property (how often it flushes) matters: too short and you are making unnecessary HTTP calls, too long and your dashboards lag behind reality.

The first instrument to reach for is a `Counter`: a number that only goes up, perfect for counting discrete events like provider calls. `Counter.builder("valuation.provider.calls").tag("provider", providerName).tag("outcome", "success").register(registry).increment()` called once after every provider call — success or failure — gives you a running total you can slice by either tag in a dashboard without writing a single line of aggregation code yourself.

The second is a `Timer`, which measures both the count and the distribution of how long something took — exactly what you want for the latency of `VehicleValuationService.valuate()` as a whole. `Timer.start(registry)` at the top of the call and `.stop(...)` when it completes records one sample per request, and because a `Timer` tracks percentiles, not just an average, you can see your p99 clearly instead of having one slow outlier hidden inside a comfortable-looking mean.

The third is a `Gauge`, and it is a different shape entirely: a `Counter` only ever increments, but a `Gauge` reports whatever the current value of something is at the moment it is sampled — the size of a queue, the number of open connections, or here, the current state of the Failsafe circuit breaker for each provider. `Gauge.builder("valuation.provider.circuit_breaker.state", breaker) { it.state.ordinal.toDouble() }.tag("provider", providerName).register(registry)` lets the registry read the breaker's live state directly, rather than you remembering to push an update every time it changes.

Every one of these examples tags by `provider` and by `outcome` — both small, fixed, finite sets of values — and never by something like VIN or request ID. That distinction is the entire game: tagging by a bounded dimension like a provider name (three possible values) turns into three time series, cheap to store and trivial to graph; tagging by something unbounded like a VIN creates a new time series per unique vehicle, which is how a metrics bill quietly explodes and how a registry starts silently dropping data under high cardinality.

Done right, this is the difference between debugging by instinct and debugging by evidence. When Carfax has a bad afternoon, nobody needs to grep logs across three services and guess at a failure rate — the `valuation.provider.calls` counter, grouped by `provider` and `outcome` on a single Datadog widget, shows "Carfax is failing 40% of calls" the moment it starts happening, and the same counter is what a monitor alerts on before a human ever notices anything is wrong.

@Component
class ProviderMetrics(private val registry: MeterRegistry) {

    fun recordProviderCall(providerName: String, outcome: String) {
        Counter.builder("valuation.provider.calls")
            .tag("provider", providerName)
            .tag("outcome", outcome)
            .description("Number of calls made to each vehicle-valuation provider")
            .register(registry)
            .increment()
    }

    suspend fun <T> timeValuationRequest(block: suspend () -> T): T {
        val sample = Timer.start(registry)
        var outcome = "error"
        try {
            val result = block()
            outcome = "success"
            return result
        } finally {
            sample.stop(
                Timer.builder("valuation.request.latency")
                    .tag("outcome", outcome)
                    .publishPercentileHistogram()
                    .register(registry)
            )
        }
    }
}

class MeteredValuationProviderClient(
    private val providerName: String,
    private val delegate: ValuationProviderClient,
    private val metrics: ProviderMetrics,
) : ValuationProviderClient {

    override suspend fun fetchValuation(vin: String): ProviderQuote {
        return try {
            val quote = delegate.fetchValuation(vin)
            metrics.recordProviderCall(providerName, "success")
            quote
        } catch (ex: Exception) {
            metrics.recordProviderCall(providerName, "failure")
            throw ex
        }
    }
}

@Component
class CircuitBreakerGauges(
    private val registry: MeterRegistry,
    private val providerBreakers: Map<String, CircuitBreaker<Any>>,
) {

    @PostConstruct
    fun registerGauges() {
        providerBreakers.forEach { (providerName, breaker) ->
            Gauge.builder("valuation.provider.circuit_breaker.state", breaker) {
                when {
                    it.isClosed -> 0.0
                    it.isHalfOpen -> 1.0
                    else -> 2.0
                }
            }
                .tag("provider", providerName)
                .description("0=closed, 1=half-open, 2=open")
                .register(registry)
        }
    }
}