Failsafe Circuit Breakers

Retries solve the problem of a provider having a brief, independent blip, but they make the wrong assumption when a provider is genuinely down for an extended period: that trying again, with a short backoff, is worth the cost. If Carfax has been failing for the last two minutes, every valuation request that arrives during that window still pays the full retry schedule — three or four attempts, each with backoff — before finally giving up, which means every caller waits the maximum possible time to receive a failure you could have predicted before making a single network call. A circuit breaker is what lets the service notice that pattern and stop paying that cost.

A `CircuitBreaker` moves through three states, and the names describe an electrical circuit on purpose. `CLOSED` is normal operation: calls flow through to the real provider, and the breaker quietly counts successes and failures as they happen. Once failures cross a configured threshold, the breaker trips `OPEN`: every call is now rejected immediately, without touching the network at all, for a fixed delay. After that delay elapses, the breaker moves to `HALF_OPEN`, a probationary state that lets a small number of trial calls through to test whether the provider has recovered — if those succeed, the breaker closes again and normal traffic resumes; if they fail, it reopens and starts the delay over.

Configuring one looks a lot like configuring the retry policy from the last lesson, just tuned to a different question — not `should this one call be retried`, but `has this provider stopped being trustworthy`. `CircuitBreaker.builder<ValuationResult>()` with `.withFailureThreshold(5, 10)` trips the breaker once 5 of the last 10 calls to Carfax have failed; `.withDelay(Duration.ofSeconds(30))` is how long it stays `OPEN` before allowing a `HALF_OPEN` trial; `.withSuccessThreshold(3)` is how many of those trial calls need to succeed before the breaker trusts the provider enough to close again. Tune the numbers per provider — a provider with tighter rate limits might want a lower failure threshold and a longer open delay than one that just occasionally times out.

An open breaker fails fast, but failing fast is still failing, and `VehicleController`'s caller does not care why the valuation is missing. This is where a `Fallback` earns its place: instead of letting a `CircuitBreakerOpenException` propagate all the way up, catch it and serve something useful instead — in this service, the Redis cache that already stores valuation lookups by VIN. A ten-minute-old Carfax valuation for this VIN is very often a better answer than no answer at all, and returning it turns a provider outage into a slightly-stale-but-correct-shaped response instead of a failed request.

Composing retry, circuit breaker, and fallback together is a single line — `Failsafe.with(fallback, circuitBreaker, retryPolicy)` — but the order of the arguments is not cosmetic. Failsafe composes the policies you pass in from outermost to innermost, in the order you list them, so this line means: fallback wraps circuit breaker, which wraps retry policy, which wraps the actual call. That nesting is what makes each policy see exactly the failures it should.

Put circuit breaker outside retry, and an open breaker rejects the call before the retry policy ever runs — no wasted attempts, no wasted backoff delays, the fallback fires almost instantly. Flip that order — retry outside circuit breaker — and every retry attempt has to individually re-enter the breaker, get rejected, and sleep through its backoff delay before trying again, which means the caller sits through several backoff sleeps in a row even though not one of those attempts ever reached the network. Circuit breaker outside retry is the ordering to reach for whenever an open breaker should mean an instant fallback rather than a slow, padded-out failure.

With this composition in place, every call `BlackBookClient`, `CarfaxClient`, and `VisClient` makes is retried for transient blips, fails fast the moment a provider is genuinely unhealthy, and falls back to a cached answer rather than an outright failure — all without a single one of those classes knowing any of this machinery exists. That closes out this module on calling third-party providers safely; the next lesson turns to a different kind of reliability question: which of Postgres, Redis, and MongoDB should actually hold which piece of this service's data.

import dev.failsafe.CircuitBreaker
import java.time.Duration

val carfaxCircuitBreaker: CircuitBreaker<ValuationResult> = CircuitBreaker.builder<ValuationResult>()
    .handle(RetryableProviderException::class.java)
    .withFailureThreshold(5, 10)       // 5 failures out of the last 10 calls trips it
    .withDelay(Duration.ofSeconds(30)) // stays OPEN for 30s before a trial call
    .withSuccessThreshold(3)           // needs 3 successful trials to CLOSE again
    .onOpen { log.warn("Carfax circuit breaker OPEN -- failing fast") }
    .onHalfOpen { log.info("Carfax circuit breaker HALF_OPEN -- testing recovery") }
    .onClose { log.info("Carfax circuit breaker CLOSED -- back to normal") }
    .build()

import dev.failsafe.Fallback

fun cachedValuationFallback(cache: ValuationCache, vin: String): Fallback<ValuationResult> =
    Fallback.builder<ValuationResult> { event ->
        // The breaker is open, or every retry was exhausted -- rather than
        // propagate the failure up to VehicleController, serve the last
        // valuation cached in Redis for this VIN. A ten-minute-old price is
        // a far better answer than a failed request.
        cache.getStaleValuation(vin) ?: throw NoStaleValuationAvailableException(vin)
    }.build()

import dev.failsafe.Failsafe
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import okhttp3.OkHttpClient
import okhttp3.Request

class CarfaxClient(
    private val client: OkHttpClient,
    private val baseUrl: String,
    private val cache: ValuationCache
) : ValuationProviderClient {

    override val providerName = "carfax"

    // Order matters: fallback is outermost so it catches anything that
    // escapes, circuitBreaker sits next so a call fails fast the instant the
    // breaker is open, and retryPolicy is innermost so only individual
    // network attempts get retried. If retryPolicy were outermost instead,
    // every retry attempt would have to re-enter the open breaker, sleep
    // through its backoff delay, and get rejected again for no benefit.
    private fun executorFor(vin: String) =
        Failsafe.with(cachedValuationFallback(cache, vin), carfaxCircuitBreaker, providerRetryPolicy)

    override suspend fun fetchValuation(vin: String): ValuationResult =
        withContext(Dispatchers.IO) {
            executorFor(vin).get {
                val request = Request.Builder()
                    .url("$baseUrl/history/valuation?vin=$vin")
                    .get()
                    .build()
                client.newCall(request).execute().use { response -> parseCarfax(vin, response) }
            }
        }
}