Failsafe Retries

The three provider clients from the last lesson are correct, but they are also naive: the first time a socket read to Carfax times out, or VIS returns a 503 because it is momentarily overloaded, `fetchValuation` just throws and the whole valuation request fails. Most of the failures these providers produce are transient — a dropped packet, a load balancer briefly routing to an unhealthy instance, a rate limiter that resets in a second — and giving up on the first sign of trouble throws away calls that would have succeeded if you had simply tried again. That is the problem a retry policy solves, and Failsafe is the library this course reaches for to build one.

Failsafe (the `dev.failsafe` library, the modern successor to the older `net.jodah:failsafe` coordinates) is a lightweight, dependency-free Java and Kotlin resilience toolkit: retries, circuit breakers, rate limiters, bulkheads, fallbacks, and timeouts, all expressed as small, composable `Policy` objects with a fluent builder API. It occupies similar territory to Resilience4j — both exist to wrap unreliable calls in configurable resilience behavior — but Failsafe's API surface is smaller and its policies compose with a single `Failsafe.with(...)` call, which is a large part of why it reads cleanly once you have more than one policy stacked together, as this module will by the next lesson.

A `RetryPolicy` is built once, up front, as an immutable, reusable object — the same instance gets shared across every call to every VIN, because a policy is just configuration, not per-call state. `RetryPolicy.builder<ValuationResult>()` lets you set `withMaxAttempts(3)` to cap how many tries a call gets in total, and `withBackoff(Duration.ofMillis(200), Duration.ofSeconds(2))` to wait longer between each attempt instead of retrying immediately. `withJitter(...)` then adds a small random offset to each backoff delay — without it, if Black Book has a blip that fails every in-flight request at once, every one of those callers would retry at exactly the same intervals, arriving back at Black Book in synchronized waves instead of a spread-out trickle.

Deciding what counts as retryable is the part that actually requires judgment. `handle(SocketTimeoutException::class.java)` or a custom `handleIf { failure -> ... }` predicate lets you say precisely which failures deserve another attempt: a `SocketTimeoutException`, a connection reset, or an HTTP 503/429 all suggest a transient condition on the other end. A 400 or 401, on the other hand, means the request itself was wrong — a malformed VIN, an expired API key — and retrying it three more times with backoff will produce the exact same rejection three more times, at the cost of three more round trips and, on some providers, three more hits against a rate limit that a client error should not be consuming in the first place.

It is tempting to treat more retries as strictly safer, but the opposite failure mode is real and has a name: a retry storm. If Black Book is genuinely struggling — not down, just slow and shedding load — every failed request retrying two or three times multiplies the traffic hitting it at precisely the moment it can least absorb more load, which can turn a partial degradation into a full outage. Retries are a tool for absorbing brief, independent blips, not for propping up a provider that is failing under sustained load; past a certain point, more attempts just add more load to something already struggling to keep up.

The safety valve for that failure mode is a retry budget: a ceiling on how many retries the whole service is allowed to spend against a given provider in a window of time, independent of how many individual calls are asking for one. Failsafe's `RetryPolicy` caps attempts per call, but a budget caps the aggregate — track retry attempts as a Datadog metric per provider, and treat a spike in that metric as a signal to back off (or let a circuit breaker, which is exactly the next lesson's subject, stop sending traffic at all) rather than letting every caller keep retrying independently forever.

Retries buy you resilience against blips, but they do nothing for a provider that is actually down for an extended stretch — in that case, every retry just delays the inevitable failure by however long the backoff schedule takes, wasting time on every single request that hits it. The next lesson adds a circuit breaker on top of this retry policy, so that once a provider's failures cross a threshold, the service stops trying it altogether — quickly, and on purpose — until there is real evidence it has recovered.

import dev.failsafe.RetryPolicy
import java.io.IOException
import java.net.SocketTimeoutException
import java.time.Duration

// A policy is stateless configuration, built once and reused across every
// call to every provider -- it is not tied to a single request.
val networkRetryPolicy: RetryPolicy<ValuationResult> = RetryPolicy.builder<ValuationResult>()
    .handle(SocketTimeoutException::class.java, IOException::class.java)
    .withBackoff(Duration.ofMillis(200), Duration.ofSeconds(2))
    .withJitter(Duration.ofMillis(100))
    .withMaxAttempts(3)
    .onRetry { event ->
        log.warn("Retrying valuation call, attempt {}", event.attemptCount)
    }
    .build()

import dev.failsafe.RetryPolicy
import java.time.Duration

class RetryableProviderException(provider: String, code: Int) :
    RuntimeException("$provider returned retryable status $code")

class NonRetryableProviderException(provider: String, code: Int) :
    RuntimeException("$provider returned non-retryable status $code")

// A 429 or a 503 usually means "try again shortly, I am just overloaded."
// A 400 means the VIN or request itself was invalid -- retrying it three
// times with backoff just burns three round trips for the same guaranteed
// failure, and on some providers counts against the rate limit besides.
val providerRetryPolicy: RetryPolicy<ValuationResult> = RetryPolicy.builder<ValuationResult>()
    .handleIf { failure -> failure is RetryableProviderException }
    .withMaxAttempts(4)
    .withBackoff(Duration.ofMillis(150), Duration.ofSeconds(3))
    .withJitter(0.25)
    .build()

import dev.failsafe.Failsafe
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import okhttp3.OkHttpClient
import okhttp3.Request

class BlackBookClient(
    private val client: OkHttpClient,
    private val baseUrl: String,
    private val retryPolicy: RetryPolicy<ValuationResult>
) : ValuationProviderClient {

    override val providerName = "black-book"

    override suspend fun fetchValuation(vin: String): ValuationResult =
        withContext(Dispatchers.IO) {
            // Failsafe wraps a plain blocking lambda -- no coroutines
            // involved -- which is exactly right here, since we are already
            // parked on an IO dispatcher thread for the duration of the call.
            Failsafe.with(retryPolicy).get {
                val request = Request.Builder().url("$baseUrl/v2/valuations/$vin").get().build()
                client.newCall(request).execute().use { response ->
                    when {
                        response.code in 500..599 || response.code == 429 ->
                            throw RetryableProviderException(providerName, response.code)
                        !response.isSuccessful ->
                            throw NonRetryableProviderException(providerName, response.code)
                        else -> parseBlackBook(vin, response)
                    }
                }
            }
        }
}