Resilience: Timeouts and Retries with Coroutines and Reactor

In any system that talks to a network, the most dangerous failure mode is not the error that comes back quickly — it is the call that never comes back at all. A downstream service that hangs will silently pin one of your threads or coroutines, exhaust your connection pool, and cascade into a full outage. The first rule of resilience is therefore to put a hard upper bound on how long any I/O operation is allowed to take. In Kotlin coroutines, that bound is expressed with `withTimeout` and `withTimeoutOrNull`, two structured-concurrency primitives that turn 'this might hang forever' into 'this finishes, one way or another, within N milliseconds.'

`withTimeout(duration) { ... }` runs the supplied block and, if it has not completed when the timer fires, cancels it and throws `TimeoutCancellationException` (a subclass of `CancellationException`). Because cancellation is cooperative, the block must contain a suspension point — a real suspend call like a coroutine HTTP request or `delay` — for the timeout to take effect; a tight CPU loop with no suspension cannot be interrupted. The sibling `withTimeoutOrNull(duration) { ... }` is the same machinery but returns `null` instead of throwing, which is the idiomatic choice when a timed-out result is simply 'no value' rather than an exceptional condition you want to propagate. Prefer the `Duration`-based overloads (`2.seconds`) over raw millis for readability.

A subtle but important detail: `TimeoutCancellationException` is a `CancellationException`, and structured concurrency treats cancellation specially. If you wrap a `withTimeout` block in a naive `try/catch (e: Exception)`, you will swallow the timeout and, worse, you may swallow legitimate cancellation of the parent scope. Catch `TimeoutCancellationException` explicitly when you want to react to a timeout, and never catch a bare `CancellationException` without rethrowing it. This keeps your coroutine well-behaved inside the cancellation tree.

Timeouts make a call fail fast; retries give a transient failure a second chance. The naive instinct is a fixed-delay loop, but constant retries hammer an already-struggling service and synchronize every client into thundering-herd waves. The standard cure is exponential backoff: each successive attempt waits roughly twice as long as the previous one, capped at some maximum. Layer jitter — a small random component added to each delay — on top so that a thousand clients that failed at the same instant do not all retry at the same instant. The result is a polite, self-throttling client that backs off precisely when the system needs breathing room.

A production-grade retry helper takes more than a count and a delay. It must decide *which* failures are retryable: a 503 or a connection reset is worth retrying, but a 400 Bad Request or a validation error will fail identically every time and should propagate immediately. It must also respect cancellation — a retry loop that catches `Throwable` will trap `CancellationException` and keep looping after its scope has been cancelled, which is a classic coroutine bug. Always rethrow `CancellationException` before deciding whether to retry, and gate retries behind a predicate so you only repeat operations that can plausibly succeed on a later attempt.

In a Spring Boot 3 codebase you frequently straddle two worlds: suspend functions in your service layer and a reactive `WebClient` returning `Mono`/`Flux` underneath. On the Reactor side, the equivalent of an exponential-backoff retry is the declarative `Retry.backoff(maxAttempts, minBackoff)` combinator passed to `retryWhen`. It supports `.maxBackoff(...)`, `.jitter(...)`, and `.filter { it is SomeRetryableException }`, mirroring everything you would build by hand in coroutines. Crucially, you can apply Reactor's timeout (`.timeout(Duration)`) and `retryWhen` on the `Mono`, then bridge the whole resilient pipeline into your suspend world with `.awaitSingle()` from `kotlinx-coroutines-reactor`, so the calling coroutine sees a single clean suspend call.

When you combine the two techniques, ordering matters. The timeout should wrap each individual attempt, not the entire retry sequence — otherwise a single global timeout could expire mid-backoff and you would lose the benefit of retrying at all. The correct shape is: retry loop on the outside, the per-attempt timeout on the inside, so every attempt gets its own fresh deadline. One subtlety: because `withTimeout` throws `TimeoutCancellationException` — itself a `CancellationException` — a retry helper that dutifully rethrows cancellation will never retry a timeout. When you *want* a timed-out attempt to be retried, use `withTimeoutOrNull` inside the block and turn the `null` into an ordinary retryable exception (for example `java.util.concurrent.TimeoutException`). Each try then fails fast if it hangs, the loop backs off, and the next try starts with a full timeout budget. If you additionally need an overall ceiling on total wall-clock time, express that as an outer `withTimeout` around the whole retry loop, giving you a two-level guarantee: bounded per-attempt latency and bounded total latency.

Finally, treat these primitives as building blocks, not a complete resilience strategy. Timeouts and retries pair naturally with a circuit breaker (to stop retrying a service that is clearly down), a bulkhead (to cap concurrent in-flight calls), and idempotency keys (because a retry of a non-idempotent write can double-charge a customer). Libraries like Resilience4j package these patterns with first-class coroutine and Reactor support. But the mental model you build here — bound the latency of every call, retry only transient failures, back off with jitter, and respect cancellation — is the foundation every higher-level tool is built on.

import kotlinx.coroutines.*
import kotlin.time.Duration.Companion.milliseconds
import kotlin.time.Duration.Companion.seconds

suspend fun slowFetch(): String {
    delay(2.seconds)            // simulate a slow downstream call
    return "payload"
}

fun main() = runBlocking {
    // Throwing variant: react to the timeout explicitly.
    val result = try {
        withTimeout(500.milliseconds) { slowFetch() }
    } catch (e: TimeoutCancellationException) {
        "fell back after timeout"
    }
    println(result)            // -> fell back after timeout

    // Null variant: a timed-out result is just 'no value'.
    val maybe: String? = withTimeoutOrNull(500.milliseconds) { slowFetch() }
    println(maybe ?: "no value")  // -> no value
}

import kotlinx.coroutines.*
import kotlin.random.Random
import kotlin.time.Duration
import kotlin.time.Duration.Companion.milliseconds
import kotlin.time.Duration.Companion.seconds

suspend fun <T> retryWithBackoff(
    maxAttempts: Int = 4,
    initialDelay: Duration = 100.milliseconds,
    maxDelay: Duration = 5.seconds,
    factor: Double = 2.0,
    retryOn: (Throwable) -> Boolean = { true },
    block: suspend () -> T,
): T {
    var currentDelay = initialDelay
    repeat(maxAttempts - 1) { attempt ->
        try {
            return block()
        } catch (e: CancellationException) {
            throw e                       // never swallow cancellation
        } catch (e: Throwable) {
            if (!retryOn(e)) throw e      // non-retryable -> propagate now
            val jitter = Random.nextLong(0, 50).milliseconds
            delay(currentDelay + jitter)
            val next = (currentDelay.inWholeMilliseconds * factor).toLong()
            currentDelay = minOf(next.milliseconds, maxDelay)
        }
    }
    return block()                        // last attempt: let it throw
}

fun main() = runBlocking {
    var calls = 0
    val value = retryWithBackoff(retryOn = { it is IllegalStateException }) {
        calls++
        if (calls < 3) error("transient failure #$calls")
        "ok on attempt $calls"
    }
    println(value)   // -> ok on attempt 3
}

import kotlinx.coroutines.withTimeoutOrNull
import org.springframework.stereotype.Service
import org.springframework.web.reactive.function.client.WebClient
import org.springframework.web.reactive.function.client.awaitBody
import java.util.concurrent.TimeoutException
import kotlin.time.Duration.Companion.seconds

@Service
class PricingClient(private val webClient: WebClient) {

    suspend fun fetchPrice(sku: String): Price =
        retryWithBackoff(
            maxAttempts = 3,
            retryOn = { it is TimeoutException || it is java.io.IOException },
        ) {
            // Timeout wraps EACH attempt -> fresh budget per retry.
            // withTimeoutOrNull turns a timeout into a plain (retryable)
            // exception instead of a CancellationException the helper rethrows.
            withTimeoutOrNull(2.seconds) {
                webClient.get()
                    .uri("/prices/{sku}", sku)
                    .retrieve()
                    .awaitBody<Price>()   // bridges Mono -> suspend
            } ?: throw TimeoutException("pricing call timed out for $sku")
        }
}

data class Price(val sku: String, val amount: Long, val currency: String)

import kotlinx.coroutines.reactor.awaitSingle
import org.springframework.web.reactive.function.client.WebClient
import org.springframework.web.reactive.function.client.bodyToMono
import reactor.util.retry.Retry
import java.time.Duration

suspend fun WebClient.fetchUser(id: String): User =
    get().uri("/users/{id}", id)
        .retrieve()
        .bodyToMono<User>()
        .timeout(Duration.ofSeconds(2))                 // per-subscription deadline
        .retryWhen(
            Retry.backoff(3, Duration.ofMillis(200))    // exponential backoff
                .maxBackoff(Duration.ofSeconds(5))
                .jitter(0.5)                            // 50% randomization
                .filter { it !is IllegalArgumentException }
        )
        .awaitSingle()                                  // Mono -> suspend

data class User(val id: String, val name: String)