Staff / Principal Engineer Guide

Advanced Backend Engineering: Kotlin & Spring Boot

A comprehensive Master Course for aspiring Staff or Principal Engineers. This guide synthesizes theoretical computer science with production-hardened patterns — from the JVM and K2 compiler to Spring Boot 4, Coroutines, Hexagonal Architecture, and distributed systems. Deep-dive preparation for high-stakes technical interviews and architectural decisions.

20Chapters

60+Code Examples

30+Interview Q&As

L5-L7Target Level

Chapter 1: The Modern JVM Ecosystem & Kotlin Philosophy

Intermediate

Kotlin is not simply "Java with less typing" — it is a pragmatic language designed to fix specific shortcomings while maintaining 100% interoperability. Understanding the JVM foundation and Kotlin's design philosophy is crucial for any senior backend interview.

1.1 The "Billion Dollar Mistake" and Null Safety

NullPointerException is the most common runtime error in traditional Java. Kotlin integrates nullability into the type system: String and String? are distinct types. At runtime there is no wrapper overhead (unlike Optional), making Kotlin's null safety performance-neutral.

kotlin

// Java — defensive checks and Optional
Optional<User> user = userRepo.findById(id);
if (user.isEmpty()) throw new NotFoundException();
return user.get();

kotlin

// Kotlin — type system enforces at compile time
val user = repository.findByName("Alice") ?: throw CustomerNotFoundException()
// String and String? are distinct types — no wrapper overhead

Interview Deep Dive

Q: How does Kotlin enforce null safety when interoperating with unannotated Java code?
A: Unannotated Java types become "Platform Types" (T!). The compiler relaxes null-checks, which is a known risk. Modern Spring Boot 4 apps use JSpecify annotations to enforce strict boundaries and eliminate platform types.

1.2 Platform Types — The Hidden Danger

When calling Java libraries without nullability annotations, Kotlin infers "platform types" (T!). These bypass null safety entirely — the compiler won't warn you, and NPEs can occur at runtime. This is the #1 source of Kotlin NPEs in real-world projects.

kotlin

// Java library without nullability annotations
public class LegacyUserService {
    public String getUserEmail(Long id) { // returns T! (platform type)
        return userDao.findEmail(id); // might return null!
    }
}

// Kotlin caller — the compiler WON'T warn you
val email: String = legacyService.getUserEmail(42) // NPE at runtime!

// Safe approach — always treat Java returns as nullable
val email: String? = legacyService.getUserEmail(42)
val safeEmail = email ?: "no-email@default.com"

// Best approach — add JSpecify annotations to Java code
@NullMarked
public class UserService {
    public @Nullable String getUserEmail(Long id) { ... }
}

Gotcha

Never trust a T! platform type. Always assign Java return values to T? (nullable) unless the Java code has @NonNull / @NullMarked annotations. Spring Framework 6+ is fully annotated with JSpecify.

1.3 Immutability and State Management

Concurrency bugs stem from shared mutable state. Use val, immutable collections (List vs MutableList), and "copy-on-write" with .copy() on data classes. Service layers should be stateless; DTOs should be immutable.

kotlin

// Anti-pattern: mutable state shared across threads
class OrderService {
    private var cachedOrders = mutableListOf<Order>() // race condition!

    fun addOrder(order: Order) {
        cachedOrders.add(order) // NOT thread-safe
    }
}

// Correct: immutable data, copy-on-write semantics
data class Order(val id: Long, val items: List<OrderItem>, val status: Status)

class OrderService(private val repository: OrderRepository) {
    fun updateStatus(order: Order, newStatus: Status): Order {
        val updated = order.copy(status = newStatus) // new object, old is untouched
        repository.save(updated)
        return updated
    }
}

*Rule of thumb: If a property is injected via constructor, use val. If it changes during the object's lifetime, ask yourself: "Should I create a new object instead?" The answer is usually yes.

1.4 The K2 Compiler and Spring Boot 4

Kotlin 2.0 introduced the K2 Compiler: a complete rewrite of the frontend compiler. It brings 2x faster compilation, smarter smart casts that work across when branches and lambda boundaries, and better type inference for builder DSLs. Spring Boot 4 leverages K2 metadata for faster startup and more efficient reflection.

kotlin

// K2 Compiler — smarter smart casts (Kotlin 2.0+)
fun processInput(input: Any) {
    if (input is String && input.length > 5) {
        // K1: requires explicit cast in some complex cases
        // K2: smart cast works across when, if-else chains, and lambdas
        println(input.uppercase()) // smart cast to String
    }
}

// K2 also enables: contracts, better type inference in builders
buildList {
    add("hello") // K2 infers MutableList<String> correctly in more cases
    add("world")
}

Interview Deep Dive

Q: What are the practical benefits of the K2 compiler for Spring Boot developers?
A: Three main benefits: (1) Build speed — K2 compiles 1.5-2x faster, significant in large monorepos. (2) Smarter smart casts — fewer explicit casts needed in complex control flow. (3) Spring Boot 4 integration — K2 metadata enables AOT (Ahead of Time) compilation for GraalVM native images, reducing startup time from seconds to milliseconds.

Chapter 2: Idiomatic Kotlin for the Spring Developer

Intermediate

Avoid the "Java developer writing Kotlin" anti-pattern. Idiomatic Kotlin code is more concise, safer, and easier to maintain. Interviewers specifically look for candidates who write Kotlin, not "Java without semicolons".

2.1 Expressions vs Statements

In Kotlin, if, when, and try are expressions — they return values. Use them to assign directly and enforce immutability. This eliminates temporary var declarations entirely.

kotlin

// Java style (anti-pattern in Kotlin)
var status: String = ""
if (response.code == 200) { status = "Success" } else { status = "Error" }

// Idiomatic Kotlin — expressions
val status = when (response.code) {
    200 -> "Success"
    404 -> "Not Found"
    in 500..599 -> "Server Error"
    else -> "Unknown"
}

2.2 Extension Functions

Extend classes without inheriting. This is Kotlin's killer feature for building clean DSLs and utility APIs. But there's a critical interview gotcha: extension functions are statically resolved.

kotlin

// Add domain-specific behavior without inheritance
fun String.toSlug(): String =
    this.lowercase()
        .replace(Regex("[^a-z0-9\s-]"), "")
        .replace(Regex("\s+"), "-")
        .trim('-')

// Usage
val slug = "My Blog Post Title!".toSlug() // "my-blog-post-title"

// Extension on a Spring class — cleaner DSL
fun ResponseEntity.BodyBuilder.jsonError(message: String): ResponseEntity<Map<String, String>> =
    this.body(mapOf("error" to message))

// IMPORTANT: extensions are resolved STATICALLY
open class Shape
class Circle : Shape()
fun Shape.name() = "Shape"
fun Circle.name() = "Circle"

val shape: Shape = Circle()
println(shape.name()) // prints "Shape" — NOT "Circle"!

Interview Trap

"What does shape.name() print when shape is actually a Circle?" — It prints "Shape" because extensions are resolved by the declared type, not runtime type. This is a classic interview question to test whether you truly understand Kotlin.

2.3 Scope Functions (let, run, with, apply, also)

The five scope functions are often confused. The key differentiator: what is the context object (this vs it) and what is the return value (the object vs the lambda result).

apply

this / Object

Configure this object. Returns the object itself. Perfect for bean initialization.

also

it / Object

Do this additional thing. Returns the object. Logging, validation.

let

it / Lambda result

Transform. Use with ?.let for null-safety.

run

this / Lambda result

Configure + compute. Returns the lambda result.

with

this / Lambda result

Group function calls on an object. Non-extension function.

kotlin

// apply — configure an object, returns the object
val user = User().apply {
    name = "Alice"
    email = "alice@example.com"
    role = Role.ADMIN
}

// let — transform + null safety, returns lambda result
val length = name?.let {
    println("Processing: $it")
    it.length
} ?: 0

// also — side effects (logging, validation), returns the object
fun createUser(request: CreateUserRequest): User =
    userRepository.save(request.toEntity())
        .also { logger.info("Created user: ${it.id}") }
        .also { eventPublisher.publish(UserCreatedEvent(it)) }

// run — configure + compute, returns lambda result
val dbConfig = config.run {
    DatabaseConfig(
        url = databaseUrl,
        poolSize = maxConnections,
        timeout = Duration.ofSeconds(connectionTimeout)
    )
}

Interview Deep Dive

Q: When would you use also vs apply?
A: Use apply when configuring properties on the object (receiver is this, so you can access properties directly). Use also for side effects like logging, where having the object as it makes the code clearer — also { logger.info("Created: $it") }.

Chapter 3: Advanced Spring Boot Architecture & Configuration

Advanced

Spring Boot 4 moves from reflection-heavy annotation scanning toward functional bean definition and AOT compilation. Understanding these internals separates senior from mid-level engineers.

3.1 The Functional Bean Definition DSL

Kotlin's beans { } DSL allows programmatic registration. Benefits: no reflection scanning at startup, compilation errors for missing dependencies, profile-aware definitions, and centralized bean configuration.

kotlin

// Beans.kt — Functional Bean Definition DSL
val applicationBeans = beans {
    bean<UserService>()
    bean {
        val repo = ref<UserRepository>()
        val config = ref<AppConfig>()
        AuthenticationService(repo, config.tokenSecret)
    }
    profile("dev") { bean<EmailSender> { MockEmailSender() } }
    profile("prod") { bean<EmailSender> { SmtpEmailSender(env.getProperty("smtp.host")!!) } }
}

// Application.kt
fun main(args: Array<String>) {
    runApplication<MyServiceApplication>(*args) {
        addInitializers(BeansInitializer(applicationBeans))
    }
}

3.2 Constructor Injection vs Field Injection

@Autowired lateinit var is an anti-pattern: it forces mutability, makes testing harder, hides dependencies, and prevents immutability guarantees. Always use Constructor Injection — Kotlin makes this elegant with primary constructor parameters.

kotlin

// Anti-pattern: field injection
@Service
class OrderService {
    @Autowired lateinit var repository: OrderRepository  // mutable!
    @Autowired lateinit var paymentClient: PaymentClient  // hidden deps!
}

// Correct: constructor injection (Kotlin makes this elegant)
@Service
class OrderService(
    private val repository: OrderRepository,
    private val paymentClient: PaymentClient
) {
    // val = immutable, thread-safe, easy to test with mocks
    // Dependencies are explicit in the constructor signature
    // No need for @Autowired — Spring infers single constructor
}

Interview Deep Dive

Q: Why is constructor injection preferred over field injection in Spring?
A: Four reasons: (1) Immutability — val guarantees thread safety. (2) Testability — pass mocks directly, no reflection needed. (3) Visibility — all dependencies are explicit in the constructor. (4) Fail-fast — missing dependencies cause startup failures, not runtime NPEs.

3.3 Configuration Properties with Immutable Data Classes

Use @ConfigurationProperties with data classes for immutable config. Once the app starts, configuration cannot change — preventing drift bugs. Spring Boot 4 uses constructor binding by default for data classes.

kotlin

// Immutable configuration with data classes
@ConfigurationProperties(prefix = "app.payment")
data class PaymentConfig(
    val apiKey: String,
    val baseUrl: String,
    val timeout: Duration = Duration.ofSeconds(30),
    val retryAttempts: Int = 3,
    val webhookSecret: String
)

// Usage — injected as a bean, immutable after startup
@Service
class PaymentService(private val config: PaymentConfig) {
    private val client = HttpClient.newBuilder()
        .connectTimeout(config.timeout)
        .build()
}

// application.yml
// app:
//   payment:
//     api-key: ${PAYMENT_API_KEY}
//     base-url: https://api.stripe.com/v1
//     webhook-secret: ${STRIPE_WEBHOOK_SECRET}

3.4 Custom Auto-Configuration

Creating custom starters is a senior-level skill. It involves @AutoConfiguration, @ConditionalOn* annotations, and the new AutoConfiguration.imports file (replacing spring.factories in Spring Boot 3+).

kotlin

// Create your own Spring Boot starter
@AutoConfiguration
@ConditionalOnClass(MetricsRegistry::class)
@EnableConfigurationProperties(MetricsProperties::class)
class MetricsAutoConfiguration {

    @Bean
    @ConditionalOnMissingBean
    fun metricsRegistry(props: MetricsProperties): MetricsRegistry =
        MetricsRegistry(props.prefix, props.tags)

    @Bean
    @ConditionalOnProperty("app.metrics.http.enabled", havingValue = "true")
    fun httpMetricsFilter(registry: MetricsRegistry): HttpMetricsFilter =
        HttpMetricsFilter(registry)
}

// META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports
// com.myapp.metrics.MetricsAutoConfiguration

*Design principle: Good auto-configurations are opinionated but overridable. Always use @ConditionalOnMissingBean so users can provide their own implementation, and @ConditionalOnProperty to let them disable features entirely.

Chapter 4: Persistence Engineering (JPA vs Exposed vs R2DBC)

Advanced

"Can I use a Kotlin data class for a JPA Entity?" No. Data classes generate equals/hashCode based on all properties — lazy-loaded @OneToMany will trigger unexpected queries. Use a standard class, mark it open, and implement equals/hashCode on the primary key only.

kotlin

@Entity
class Customer(
    @Column(nullable = false)
    var name: String,
) {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    var id: Long? = null

    // CRITICAL: equals/hashCode on business key or ID only
    override fun equals(other: Any?): Boolean {
        if (this === other) return true
        if (other !is Customer) return false
        return id != null && id == other.id
    }
    override fun hashCode(): Int = 14592834 // constant — safe for sets
}

Common Mistake

Using data class for JPA entities causes: (1) lazy loading triggers in toString(), (2) equals/hashCode on all fields including lazy collections, (3) copy() creates detached entities that confuse Hibernate session. Always use regular class for JPA entities.

4.1 Exposed — Type-Safe SQL DSL

JetBrains' Exposed library takes a different approach: instead of mapping objects to tables (ORM), it provides a type-safe DSL to write SQL. Queries are validated at compile time — no more JPQL string typos discovered in production.

kotlin

// Exposed — type-safe SQL DSL (no ORM magic)
object Users : LongIdTable("users") {
    val name = varchar("name", 255)
    val email = varchar("email", 255).uniqueIndex()
    val createdAt = datetime("created_at").defaultExpression(CurrentDateTime)
}

// Type-safe queries — compile-time errors for wrong column types
suspend fun findActiveUsers(minAge: Int): List<UserDto> = newSuspendedTransaction {
    Users.selectAll()
        .where { Users.name.isNotNull() }
        .orderBy(Users.createdAt, SortOrder.DESC)
        .limit(100)
        .map { row ->
            UserDto(
                id = row[Users.id].value,
                name = row[Users.name],
                email = row[Users.email]
            )
        }
}

4.2 R2DBC — Reactive Persistence

For fully non-blocking applications, R2DBC provides reactive database access. Combined with Kotlin coroutines, you get suspend functions and Flow for streaming query results with backpressure.

kotlin

// Reactive persistence with R2DBC + Coroutines
@Repository
class UserRepository(private val client: DatabaseClient) {

    suspend fun findById(id: Long): User? =
        client.sql("SELECT * FROM users WHERE id = :id")
            .bind("id", id)
            .map { row, _ -> row.toUser() }
            .awaitOneOrNull()

    fun findAll(): Flow<User> =
        client.sql("SELECT * FROM users ORDER BY created_at DESC")
            .map { row, _ -> row.toUser() }
            .flow()

    suspend fun save(user: User): User =
        client.sql("""
            INSERT INTO users (name, email) VALUES (:name, :email)
            RETURNING *
        """.trimIndent())
            .bind("name", user.name)
            .bind("email", user.email)
            .map { row, _ -> row.toUser() }
            .awaitOne()
}

JPA vs Exposed vs R2DBC

Feature	Spring Data JPA	Exposed	R2DBC
Philosophy	ORM (object-centric)	SQL DSL (data-centric)	Reactive (stream-centric)
Performance	Proxies, cache overhead	Lightweight, direct SQL	Non-blocking, high throughput
Async Support	Blocking JDBC	Coroutines (native)	Reactive / Coroutines
Type Safety	JPQL strings	Kotlin DSL (compile-time)	Raw SQL strings
Learning Curve	High (Hibernate quirks)	Medium (SQL knowledge)	High (reactive paradigm)
Best For	CRUD-heavy, existing teams	Complex queries, Kotlin-first	High-concurrency services

Interview Deep Dive

Q: When would you choose Exposed over JPA for a new Kotlin project?
A: Choose Exposed when: (1) your team has strong SQL skills, (2) you need complex queries with joins, CTEs, or window functions, (3) you want compile-time query validation, (4) you're building a Kotlin-first codebase (no Java team to support). Stick with JPA when migrating existing Java projects or when the team is already proficient with Hibernate.

Chapter 5: Asynchronous Systems (Coroutines vs Virtual Threads)

Advanced

Coroutines: lightweight threads where suspend functions become state machines, releasing the thread while waiting. Virtual Threads (Java 21+): better for migrating legacy blocking apps without code changes. For new Kotlin projects, Coroutines remain superior thanks to Flow, select, structured concurrency, and cancellation.

5.1 Suspend Functions & Spring WebFlux

Spring Boot natively supports suspend functions in controllers. When a function suspends, the underlying thread is released back to the pool. The Flow return type enables streaming responses with backpressure.

kotlin

@RestController
class UserController(private val service: UserService) {

    @GetMapping("/users/{id}")
    suspend fun getUser(@PathVariable id: Long): UserDto {
        return service.getUserById(id)
    }

    @GetMapping("/users")
    fun getAllUsers(): Flow<UserDto> {
        return service.getAllUsers() // returns Flow, not List — backpressure-ready
    }
}

5.2 Structured Concurrency & Parallel Calls

The killer feature of coroutines is structured concurrency: if one async call fails, all siblings are automatically cancelled. No orphan threads, no resource leaks. Compare this to CompletableFuture where you must manually track and cancel.

kotlin

// Coroutines — structured concurrency, rich operators
suspend fun fetchUserDashboard(userId: Long): Dashboard = coroutineScope {
    val profile = async { userService.getProfile(userId) }
    val orders = async { orderService.getRecentOrders(userId) }
    val recommendations = async { mlService.getRecommendations(userId) }

    Dashboard(
        profile = profile.await(),
        orders = orders.await(),
        recommendations = recommendations.await()
    )
    // If any async fails, ALL are cancelled (structured concurrency)
}

// Flow — reactive streams with backpressure
fun streamPrices(symbol: String): Flow<Price> = flow {
    while (currentCoroutineContext().isActive) {
        emit(priceService.getLatestPrice(symbol))
        delay(100) // non-blocking, releases thread
    }
}
    .conflate()           // drop intermediate if consumer is slow
    .distinctUntilChanged() // skip duplicates
    .catch { e -> logger.error("Price stream error", e) }

Interview Deep Dive

Q: If Java 21+ has Virtual Threads, do we still need Coroutines?
A: Yes. Coroutines offer: (1) Rich operators — Flow (map, filter, conflate), select, channels. (2) Structured cancellation — parent-child scope hierarchy. (3) Explicit suspension points — suspend keyword makes it clear where thread-release happens. Virtual Threads are ideal for migrating legacy blocking code (JDBC, file I/O) without rewriting.

5.3 Channels — Producer/Consumer Pattern

Channels provide a CSP-style (Communicating Sequential Processes) mechanism for coroutines. They support fan-out (multiple consumers), fan-in (multiple producers), and backpressure via bounded buffers.

kotlin

// Channels — producer/consumer pattern
val orderChannel = Channel<Order>(capacity = 100)

// Producer coroutine
launch {
    orderStream.collect { order ->
        orderChannel.send(order) // suspends if buffer full (backpressure)
    }
}

// Multiple consumers (fan-out)
repeat(4) { workerId ->
    launch {
        for (order in orderChannel) {
            processOrder(order) // each order processed by exactly one worker
            logger.info("Worker $workerId processed order ${order.id}")
        }
    }
}

Feature	Coroutines	Virtual Threads
Paradigm	Cooperative (suspend/resume)	Preemptive (OS-like)
Cancellation	Structured, hierarchical	Thread.interrupt() (legacy)
Streaming	Flow (backpressure built-in)	No native equivalent
Blocking I/O	Requires Dispatchers.IO	Works natively (pin-free)
Best For	New Kotlin projects, reactive	Legacy Java migration

Chapter 6: Hexagonal Architecture (Ports & Adapters)

Expert

Dependencies point inwards. The Domain knows nothing about the Web or Database. This is the most asked architecture pattern in Staff+ interviews. The core insight: your business logic should be testable without Spring, without a database, without HTTP.

Presentation Layer (REST Controllers, GraphQL)

Adapters IN — receive external requests

depends on

Application Layer (Use Cases, Command Handlers)

Orchestration — coordinates domain + ports

depends on

Domain Layer (Entities, Value Objects, Domain Services)

Pure business logic — NO framework dependencies

implemented by

Infrastructure Layer (JPA Repos, HTTP Clients, Kafka)

Adapters OUT — implement domain ports

6.1 Defining Ports (Interfaces)

Ports are interfaces defined in the domain layer. Input ports (use cases) define what the application can do. Output ports define what the application needs from infrastructure.

kotlin

// domain/port/in — Use cases (what the app CAN do)
interface CreateOrderUseCase {
    suspend fun execute(command: CreateOrderCommand): Order
}

// domain/port/out — Infrastructure contracts (what the app NEEDS)
interface OrderRepository {
    suspend fun save(order: Order): Order
    suspend fun findById(id: OrderId): Order?
}

interface PaymentGateway {
    suspend fun charge(amount: Money, method: PaymentMethod): PaymentResult
}

interface EventPublisher {
    suspend fun publish(event: DomainEvent)
}

6.2 Rich Domain Model

Domain entities contain business rules — not just data. Use private constructor with factory methods to enforce invariants. Business validations live in the entity, not in the service layer.

kotlin

// domain/model — Pure Kotlin, NO framework annotations
data class Order private constructor(
    val id: OrderId,
    val customerId: CustomerId,
    val items: List<OrderItem>,
    val status: OrderStatus,
    val total: Money,
    val createdAt: Instant
) {
    // Business rules live HERE, not in the service
    fun cancel(): Order {
        require(status == OrderStatus.PENDING) {
            "Can only cancel pending orders, current: $status"
        }
        return copy(status = OrderStatus.CANCELLED)
    }

    fun addItem(item: OrderItem): Order {
        require(status == OrderStatus.DRAFT) { "Cannot modify non-draft order" }
        return copy(
            items = items + item,
            total = total + item.price * item.quantity
        )
    }

    companion object {
        fun create(customerId: CustomerId, items: List<OrderItem>): Order {
            require(items.isNotEmpty()) { "Order must have at least one item" }
            return Order(
                id = OrderId.generate(),
                customerId = customerId,
                items = items,
                status = OrderStatus.DRAFT,
                total = items.sumOf { it.price * it.quantity },
                createdAt = Instant.now()
            )
        }
    }
}

Interview Deep Dive

Q: Why use private constructor with a companion object factory?
A: It enforces invariants at construction time. Every Order that exists in your system is guaranteed to have at least one item because the create() factory validates this. You can never accidentally create an invalid Order — the compiler prevents it.

6.3 Application Services (Use Case Orchestration)

Services orchestrate the domain model and ports. They do NOT contain business logic — that belongs in the domain. Services handle: transaction boundaries, port coordination, and event publishing.

kotlin

// domain/service — Orchestrates ports, NO infrastructure awareness
@Service
class CreateOrderService(
    private val orderRepo: OrderRepository,
    private val paymentGateway: PaymentGateway,
    private val eventPublisher: EventPublisher
) : CreateOrderUseCase {

    override suspend fun execute(command: CreateOrderCommand): Order {
        val order = Order.create(command.customerId, command.items)
        val savedOrder = orderRepo.save(order)

        val paymentResult = paymentGateway.charge(order.total, command.paymentMethod)

        val confirmedOrder = when (paymentResult) {
            is PaymentResult.Success -> savedOrder.copy(status = OrderStatus.CONFIRMED)
            is PaymentResult.Declined -> savedOrder.copy(status = OrderStatus.PAYMENT_FAILED)
        }

        orderRepo.save(confirmedOrder)
        eventPublisher.publish(OrderCreatedEvent(confirmedOrder))
        return confirmedOrder
    }
}

6.4 Adapters (Pluggable Infrastructure)

Adapters implement ports. Input adapters (controllers) call use cases. Output adapters (repositories, HTTP clients) implement domain port interfaces. You can swap PostgreSQL for MongoDB by changing only the adapter — zero domain changes.

kotlin

// infrastructure/adapter/in — REST adapter (plugs into the port)
@RestController
@RequestMapping("/api/orders")
class OrderController(private val createOrder: CreateOrderUseCase) {

    @PostMapping
    suspend fun create(@RequestBody request: CreateOrderRequest): ResponseEntity<OrderResponse> {
        val command = request.toCommand() // map DTO to domain command
        val order = createOrder.execute(command)
        return ResponseEntity.status(201).body(order.toResponse())
    }
}

// infrastructure/adapter/out — JPA adapter (implements the port)
@Repository
class JpaOrderRepository(
    private val jpa: SpringDataOrderRepository
) : OrderRepository {

    override suspend fun save(order: Order): Order {
        val entity = order.toEntity()
        val saved = jpa.save(entity)
        return saved.toDomain()
    }

    override suspend fun findById(id: OrderId): Order? =
        jpa.findById(id.value)?.toDomain()
}

Directory Structure

domain/ — model (entities, value objects), port (in/out interfaces), service (domain services)
application/ — use case implementations, command/query handlers
infrastructure/ — adapter/in (REST, gRPC), adapter/out (JPA, HTTP clients, Kafka), config

*The litmus test: Can you run your domain's unit tests without starting Spring, without a database, without any external service? If yes, your architecture is properly hexagonal. If not, you have infrastructure leaking into your domain.

Chapter 7: Distributed Systems Patterns

Expert

Distributed systems are where theory meets reality. You need to understand: idempotency, circuit breakers, saga patterns, CQRS, and context propagation. These are the topics that separate Staff from Senior engineers.

7.1 Idempotency

Idempotency: Client sends Idempotency-Key header. Server checks Redis: key exists & completed returns cached result; key exists & processing returns 409 Conflict; key missing means process and cache. This prevents duplicate charges, duplicate emails, duplicate orders.

kotlin

suspend fun processPayment(key: String, request: PaymentRequest): PaymentResponse {
    val lock = redisTemplate.opsForValue()
        .setIfAbsent(key, "PROCESSING", Duration.ofMinutes(5))
    if (lock == false) throw DuplicateRequestException()

    return try {
        val result = paymentGateway.charge(request)
        redisTemplate.opsForValue().set(key, result, Duration.ofHours(24))
        result
    } catch (e: Exception) {
        redisTemplate.delete(key)
        throw e
    }
}

7.2 Circuit Breaker Pattern

When a downstream service is failing, continuing to send requests makes things worse (cascading failures). A circuit breaker "opens" after a threshold of failures, rejecting calls immediately. After a cooldown period, it enters "half-open" state to test recovery.

kotlin

// Resilience4j with Kotlin coroutines
@Service
class ProductClient(
    private val webClient: WebClient,
    private val circuitBreakerRegistry: CircuitBreakerRegistry
) {
    private val cb = circuitBreakerRegistry.circuitBreaker("productService") {
        failureRateThreshold(50f)
        waitDurationInOpenState(Duration.ofSeconds(30))
        slidingWindowSize(10)
        minimumNumberOfCalls(5)
    }

    suspend fun getProduct(id: Long): Product =
        cb.executeSuspendFunction {
            webClient.get()
                .uri("/products/{id}", id)
                .retrieve()
                .awaitBody<Product>()
        }
}

// States: CLOSED (normal) -> OPEN (failing, reject calls) -> HALF_OPEN (test)

Interview Deep Dive

Q: How do you decide the circuit breaker threshold and window?
A: Start with production metrics: (1) Failure rate threshold — typically 50% over a sliding window of 10-20 calls. (2) Wait duration — match the downstream service's typical recovery time (30s-60s). (3) Minimum calls — at least 5-10 to avoid opening on a single failure. Always combine with retries (with exponential backoff) and fallback responses.

7.3 Saga Pattern

Distributed transactions (2PC) are fragile and slow. The Saga pattern breaks a transaction into a sequence of local transactions, each with a compensating action for rollback. Two flavors: orchestration (central coordinator) and choreography (event-driven).

kotlin

// Saga — distributed transaction via compensating actions
class OrderSaga(
    private val orderService: OrderService,
    private val paymentService: PaymentService,
    private val inventoryService: InventoryService,
    private val shippingService: ShippingService
) {
    suspend fun execute(command: CreateOrderCommand): OrderResult {
        val order = orderService.create(command)

        val payment = try {
            paymentService.charge(order)
        } catch (e: Exception) {
            orderService.cancel(order.id) // compensate
            throw e
        }

        val reservation = try {
            inventoryService.reserve(order.items)
        } catch (e: Exception) {
            paymentService.refund(payment.id) // compensate
            orderService.cancel(order.id)     // compensate
            throw e
        }

        return OrderResult(order, payment, reservation)
    }
}

Design Consideration

Sagas introduce eventual consistency. Between steps, the system is in an intermediate state. Your UI must handle this: show "Processing..." states, use polling or WebSockets for status updates, and design for the possibility that compensating actions may also fail.

7.4 CQRS (Command Query Responsibility Segregation)

Separate the write model (rich domain, normalized) from the read model (denormalized projections, optimized for queries). This is essential at scale: writes go to a transactional DB, reads come from materialized views or search indices (Elasticsearch).

kotlin

// CQRS — separate read and write models
// Write side: rich domain model
@Service
class OrderCommandService(
    private val repository: OrderRepository,
    private val eventStore: EventStore
) {
    suspend fun handle(cmd: PlaceOrderCommand): OrderId {
        val order = Order.create(cmd.customerId, cmd.items)
        repository.save(order)
        eventStore.append(OrderPlacedEvent(order))
        return order.id
    }
}

// Read side: optimized projection for queries
@Service
class OrderQueryService(private val readDb: OrderReadRepository) {
    suspend fun getOrderSummary(orderId: OrderId): OrderSummaryDto =
        readDb.findSummaryById(orderId) // denormalized, fast read
            ?: throw OrderNotFoundException(orderId)

    fun getOrdersByCustomer(customerId: CustomerId): Flow<OrderListItemDto> =
        readDb.streamByCustomerId(customerId) // pre-joined, no N+1
}

7.5 Context Propagation

In async/coroutine systems, ThreadLocal (e.g. MDC for logging) breaks because coroutines hop between threads. Use Micrometer Context Propagation to ensure Trace IDs and Span IDs flow correctly through suspend functions and reactive chains.

kotlin

// Coroutine context propagation for tracing
val mdcContext = MDCContext() // carries MDC (trace IDs) across suspensions

suspend fun processRequest(traceId: String) {
    withContext(mdcContext + Dispatchers.IO) {
        MDC.put("traceId", traceId)
        logger.info("Processing...") // traceId is available in logs

        val result = async { externalService.call() } // traceId propagated!
        result.await()
    }
}

// application.yml — automatic propagation with Micrometer
// spring:
//   reactor:
//     context-propagation: auto
// management:
//   tracing:
//     sampling:
//       probability: 1.0

Retry with Backoff

Exponential backoff + jitter. Never retry non-idempotent operations. Use circuit breaker to stop retries when service is down.

Outbox Pattern

Write events to a DB "outbox" table in the same transaction as the business data. A separate poller publishes to Kafka — guarantees at-least-once delivery.

Bulkhead

Isolate resources per service/consumer. One slow downstream can't exhaust all threads. Use separate thread pools or Semaphore-based limits.

Dead Letter Queue

Messages that fail processing N times go to a DLQ for manual inspection. Critical for debugging production issues without losing data.

Chapter 8: Testing Strategies for Production Systems

Advanced

Testing is not optional for Staff-level engineers — it's a core competency. You need to know when to use unit tests vs integration tests vs contract tests, how to mock Kotlin-specific constructs, and how to test without a database vs with a real one.

E2E Tests

Slow, brittle, few

Integration Tests (Testcontainers)

Real DB, real queues, moderate count

Unit Tests (MockK, JUnit 5)

Fast, focused, many — test business logic here

8.1 MockK — Kotlin-First Mocking

MockK is the de facto mocking library for Kotlin. Unlike Mockito, it handles final classes (all Kotlin classes are final by default), extension functions, suspend functions, and object declarations natively. Use coEvery / coVerify for coroutine functions.

kotlin

// MockK — Kotlin-first mocking (handles final classes, suspend, extensions)
class OrderServiceTest {

    private val repository = mockk<OrderRepository>()
    private val paymentGateway = mockk<PaymentGateway>()
    private val eventPublisher = mockk<EventPublisher>(relaxUnitFun = true)
    private val service = CreateOrderService(repository, paymentGateway, eventPublisher)

    @Test
    fun `should create order and charge payment`() = runTest {
        // Given
        val command = createOrderCommand()
        coEvery { repository.save(any()) } returnsArgument 0
        coEvery { paymentGateway.charge(any(), any()) } returns PaymentResult.Success("tx-123")

        // When
        val order = service.execute(command)

        // Then
        assertThat(order.status).isEqualTo(OrderStatus.CONFIRMED)
        coVerify(exactly = 2) { repository.save(any()) } // draft + confirmed
        coVerify { eventPublisher.publish(any<OrderCreatedEvent>()) }
    }

    @Test
    fun `should fail order when payment is declined`() = runTest {
        coEvery { repository.save(any()) } returnsArgument 0
        coEvery { paymentGateway.charge(any(), any()) } returns PaymentResult.Declined("insufficient funds")

        val order = service.execute(createOrderCommand())

        assertThat(order.status).isEqualTo(OrderStatus.PAYMENT_FAILED)
    }
}

8.2 Testcontainers — Real Infrastructure

Never mock the database. Testcontainers spins up real PostgreSQL, Redis, Kafka containers for integration tests. This catches real issues: FK constraints, unique index violations, SQL dialect differences, and N+1 queries.

kotlin

// Testcontainers — real database in integration tests
@Testcontainers
@SpringBootTest
class UserRepositoryIntegrationTest {

    companion object {
        @Container
        val postgres = PostgreSQLContainer("postgres:16-alpine").apply {
            withDatabaseName("testdb")
            withUsername("test")
            withPassword("test")
        }

        @JvmStatic
        @DynamicPropertySource
        fun properties(registry: DynamicPropertyRegistry) {
            registry.add("spring.datasource.url") { postgres.jdbcUrl }
            registry.add("spring.datasource.username") { postgres.username }
            registry.add("spring.datasource.password") { postgres.password }
        }
    }

    @Autowired
    lateinit var repository: UserRepository

    @Test
    fun `should enforce unique email constraint`() {
        repository.save(User(name = "Alice", email = "alice@test.com"))

        assertThrows<DataIntegrityViolationException> {
            repository.save(User(name = "Bob", email = "alice@test.com"))
        }
    }

    @Test
    fun `should cascade delete orders when user is deleted`() {
        val user = repository.save(User(name = "Charlie", email = "charlie@test.com"))
        orderRepository.save(Order(userId = user.id!!, total = 99.99))

        repository.deleteById(user.id!!)

        assertThat(orderRepository.findByUserId(user.id!!)).isEmpty()
    }
}

Interview Deep Dive

Q: When would you mock the database vs use Testcontainers?
A: Mock (unit test) when testing business logic — the domain service's behavior given certain repository responses. Use Testcontainers (integration test) when testing data access logic — that queries return correct results, constraints are enforced, and migrations work. Never mock in integration tests; never use real DBs in unit tests.

8.3 Architecture Tests with ArchUnit

Enforce architectural rules as automated tests. Prevent domain-from-depending-on-infrastructure drift, ensure naming conventions, and validate layer boundaries — all checked in CI.

kotlin

// ArchUnit — enforce architecture rules at compile-time
@AnalyzeClasses(packages = ["com.myapp"])
class ArchitectureTest {

    @ArchTest
    val domainShouldNotDependOnInfrastructure: ArchRule =
        noClasses()
            .that().resideInAPackage("..domain..")
            .should().dependOnClassesThat().resideInAPackage("..infrastructure..")
            .because("Domain must not know about infrastructure")

    @ArchTest
    val controllersShouldNotAccessRepositoriesDirectly: ArchRule =
        noClasses()
            .that().resideInAPackage("..adapter.in..")
            .should().dependOnClassesThat().resideInAPackage("..adapter.out..")
            .because("Input adapters should use domain ports, not output adapters")

    @ArchTest
    val servicesShouldBeSuffixed: ArchRule =
        classes()
            .that().implement(UseCase::class.java)
            .should().haveSimpleNameEndingWith("Service")
}

*Testing strategy for hexagonal architecture: Domain layer = unit tests with MockK (fast, no Spring). Application layer = integration tests with @SpringBootTest + Testcontainers. Infrastructure adapters = contract tests verifying port implementations. Architecture = ArchUnit tests in CI.

Chapter 9: The Senior Interview Gauntlet

Expert

This chapter combines everything: system design, live coding, and behavioral questions. These are the exact scenarios you'll face in L5-L7 backend interviews at top tech companies.

9.1 System Design: High-Throughput Notification Service

Design a notification service that sends millions of notifications (email, push, SMS) with guarantees: at-least-once delivery, idempotency, rate limiting per user, and multi-channel support.

Key Design Decisions

Message Queue: Kafka for high throughput + ordering by partition key (user ID).
Idempotency: Redis-based dedup with TTL (as shown in Ch.7).
Rate Limiting: Per-user token bucket to prevent notification fatigue.
Pagination: Keyset pagination (not OFFSET) for querying millions of users.
Retry: DLQ for failed sends, exponential backoff per channel.

kotlin

// Notification Service — High-Throughput Design
@Service
class NotificationService(
    private val kafka: KafkaTemplate<String, NotificationEvent>,
    private val rateLimiter: RateLimiter,
    private val templateEngine: TemplateEngine
) {
    suspend fun sendBulkNotification(
        userIds: List<Long>,
        template: NotificationTemplate
    ) {
        // Keyset pagination — no OFFSET, scales to millions
        var lastId = 0L
        do {
            val batch = userRepository.findNextBatch(afterId = lastId, limit = 1000)
            batch.forEach { user ->
                val event = NotificationEvent(
                    userId = user.id,
                    channel = user.preferredChannel,
                    content = templateEngine.render(template, user),
                    idempotencyKey = "${template.id}-${user.id}"
                )
                kafka.send("notifications", user.id.toString(), event)
            }
            lastId = batch.lastOrNull()?.id ?: break
        } while (batch.size == 1000)
    }
}

9.2 Live Coding: Token Bucket Rate Limiter

Implement the Token Bucket algorithm with Kotlin Coroutines Mutex. This is a classic interview problem — interviewers want to see thread-safe code, mathematical correctness, and clean API design.

kotlin

// Token Bucket Rate Limiter with Coroutines
class RateLimiter(val capacity: Int, val refillRate: Int) {
    private val mutex = Mutex()
    private var tokens = capacity.toDouble()
    private var lastRefill = System.currentTimeMillis()

    suspend fun tryAcquire(): Boolean {
        mutex.withLock {
            refill()
            if (tokens >= 1) { tokens -= 1; return true }
            return false
        }
    }

    private fun refill() {
        val now = System.currentTimeMillis()
        val delta = (now - lastRefill) / 1000.0
        tokens = min(capacity.toDouble(), tokens + delta * refillRate)
        lastRefill = now
    }
}

// Usage in a filter
class RateLimitFilter(private val limiter: RateLimiter) : WebFilter {
    override fun filter(exchange: ServerWebExchange, chain: WebFilterChain): Mono<Void> =
        mono {
            if (!limiter.tryAcquire()) {
                exchange.response.statusCode = HttpStatus.TOO_MANY_REQUESTS
                exchange.response.setComplete().awaitFirstOrNull()
            } else {
                chain.filter(exchange).awaitFirstOrNull()
            }
        }
}

9.3 Rapid-Fire Interview Questions

Question 1

Q: What is the difference between coroutineScope and supervisorScope?
A: coroutineScope: if any child fails, all siblings are cancelled (fail-fast). supervisorScope: child failures don't affect siblings — each is independent. Use supervisorScope for fan-out operations where partial results are acceptable (e.g., fetching data from multiple optional sources).

Question 2

Q: Explain the N+1 query problem and how to solve it in JPA.
A: Loading a parent entity and then lazily loading each child individually. Solutions: (1) @EntityGraph for declarative fetch joins, (2) JOIN FETCH in JPQL, (3) @BatchSize to batch lazy loads. In Exposed: just write the JOIN explicitly in the DSL.

Question 3

Q: How do you handle database migrations in a zero-downtime deployment?
A: Expand-and-contract pattern: (1) Add new column (nullable, with default). (2) Deploy code that writes to both old and new columns. (3) Backfill existing data. (4) Deploy code that reads from new column only. (5) Drop old column. Never rename or drop columns in a single deployment.

Question 4

Q: What is backpressure and how do Kotlin Flows handle it?
A: Backpressure occurs when a producer emits data faster than a consumer can process it. Flows handle it natively because emit() suspends until the collector is ready. Additional strategies: conflate() drops intermediate values, buffer() adds a buffer, and collectLatest cancels slow processing when new values arrive.

Question 5

Q: When would you use event sourcing vs traditional CRUD?
A: Event sourcing when: (1) you need a complete audit trail (finance, healthcare), (2) you need to replay events to rebuild state or create new projections, (3) domain events drive other systems (CQRS). Traditional CRUD when: (1) simple CRUD operations, (2) team isn't familiar with event-driven patterns, (3) eventual consistency is unacceptable for the business.

Question 6

Q: How does Kotlin's inline modifier work and when should you use it?
A: inline replaces the function call with the function body at the call site — no lambda object allocation, no virtual dispatch. Use it for: (1) higher-order functions called in tight loops (performance), (2) reified type parameters (inline fun <reified T> — type info available at runtime). Don't inline large functions — it increases bytecode size.

"Why Kotlin?" Focus on correctness (null safety), maintainability (expressiveness), and performance (Coroutines). Avoid subjective answers like "it's nicer" — cite specific technical advantages with production impact.

Chapter 10: API Design & REST Best Practices

Advanced

A well-designed API is the most important interface in your system. It outlasts any implementation. Senior engineers are expected to design APIs that are consistent, evolvable, and self-documenting. This chapter covers RESTful design, error handling with ProblemDetail, versioning strategies, and cursor-based pagination.

10.1 RESTful Controller Design

Use proper HTTP methods: GET for reads, POST for creation (returns 201 Created with Location header), PATCH for partial updates, DELETE returns 204 No Content. Always return the created/updated resource in the response body.

kotlin

// RESTful API with proper HTTP semantics
@RestController
@RequestMapping("/api/v1/products")
class ProductController(
    private val productService: ProductService,
    private val assembler: ProductModelAssembler
) {
    @GetMapping
    suspend fun list(
        @RequestParam(defaultValue = "0") page: Int,
        @RequestParam(defaultValue = "20") size: Int,
        @RequestParam(required = false) category: String?
    ): ResponseEntity<PagedModel<ProductResponse>> {
        val products = productService.findAll(
            PageRequest.of(page, size, Sort.by("createdAt").descending()),
            category
        )
        return ResponseEntity.ok(assembler.toPagedModel(products))
    }

    @PostMapping
    suspend fun create(
        @Valid @RequestBody request: CreateProductRequest
    ): ResponseEntity<ProductResponse> {
        val product = productService.create(request.toCommand())
        val uri = URI.create("/api/v1/products/${product.id}")
        return ResponseEntity.created(uri).body(product.toResponse())
    }

    @PatchMapping("/{id}")
    suspend fun update(
        @PathVariable id: Long,
        @Valid @RequestBody request: UpdateProductRequest
    ): ResponseEntity<ProductResponse> {
        val product = productService.update(id, request.toCommand())
        return ResponseEntity.ok(product.toResponse())
    }

    @DeleteMapping("/{id}")
    @ResponseStatus(HttpStatus.NO_CONTENT)
    suspend fun delete(@PathVariable id: Long) {
        productService.delete(id)
    }
}

10.2 Error Handling with RFC 9457 Problem Detail

Spring Boot 3+ natively supports ProblemDetail (RFC 9457) — a standardized JSON error format. Stop returning ad-hoc error objects. Use @RestControllerAdvice with typed exception handlers. Include a type URI for documentation, and setProperty() for domain-specific fields.

kotlin

// RFC 9457 Problem Detail — structured error responses
@RestControllerAdvice
class GlobalExceptionHandler : ResponseEntityExceptionHandler() {

    @ExceptionHandler(OrderNotFoundException::class)
    fun handleNotFound(ex: OrderNotFoundException): ProblemDetail =
        ProblemDetail.forStatusAndDetail(
            HttpStatus.NOT_FOUND,
            ex.message ?: "Resource not found"
        ).apply {
            title = "Order Not Found"
            type = URI.create("https://api.myapp.com/errors/order-not-found")
            setProperty("orderId", ex.orderId)
        }

    @ExceptionHandler(BusinessRuleViolation::class)
    fun handleBusinessRule(ex: BusinessRuleViolation): ProblemDetail =
        ProblemDetail.forStatusAndDetail(
            HttpStatus.UNPROCESSABLE_ENTITY,
            ex.message
        ).apply {
            title = "Business Rule Violation"
            type = URI.create("https://api.myapp.com/errors/business-rule")
            setProperty("rule", ex.ruleName)
        }

    @ExceptionHandler(ConstraintViolationException::class)
    fun handleValidation(ex: ConstraintViolationException): ProblemDetail =
        ProblemDetail.forStatusAndDetail(
            HttpStatus.BAD_REQUEST,
            "Validation failed"
        ).apply {
            title = "Validation Error"
            setProperty("violations", ex.constraintViolations.map {
                mapOf("field" to it.propertyPath.toString(), "message" to it.message)
            })
        }
}

// Response format:
// {
//   "type": "https://api.myapp.com/errors/order-not-found",
//   "title": "Order Not Found",
//   "status": 404,
//   "detail": "Order with ID 42 does not exist",
//   "orderId": 42
// }

Interview Deep Dive

Q: How do you design error responses for a public API?
A: Use RFC 9457 ProblemDetail: consistent structure with type (URI to error docs), title (human-readable), status (HTTP code), detail (instance-specific message), and custom extension fields. Never expose stack traces or internal details. Map internal exceptions to business-friendly error codes.

10.3 API Versioning

Three strategies: URI path (/api/v1/) — most explicit, easy to route. Header-based (X-API-Version) — cleaner URLs, harder to test. Content negotiation (Accept: application/vnd.myapi.v2+json) — RESTful purist approach. For most teams, URI path versioning wins on simplicity.

kotlin

// API Versioning — URI path (most explicit, widely adopted)
@RestController
@RequestMapping("/api/v1/users")
class UserControllerV1(private val service: UserService) {
    @GetMapping("/{id}")
    suspend fun getUser(@PathVariable id: Long): UserResponseV1 =
        service.findById(id).toResponseV1()
}

@RestController
@RequestMapping("/api/v2/users")
class UserControllerV2(private val service: UserService) {
    @GetMapping("/{id}")
    suspend fun getUser(@PathVariable id: Long): UserResponseV2 =
        service.findById(id).toResponseV2() // includes new fields
}

// API Versioning — Header-based (cleaner URLs)
@RestController
@RequestMapping("/api/users")
class UserController(private val service: UserService) {
    @GetMapping("/{id}", headers = ["X-API-Version=1"])
    suspend fun getUserV1(@PathVariable id: Long): UserResponseV1 =
        service.findById(id).toResponseV1()

    @GetMapping("/{id}", headers = ["X-API-Version=2"])
    suspend fun getUserV2(@PathVariable id: Long): UserResponseV2 =
        service.findById(id).toResponseV2()
}

10.4 Cursor-Based Pagination

OFFSET pagination is O(n) — the database scans and discards all previous rows. Keyset pagination uses a cursor (typically a timestamp or ID) and seeks directly to the position. This is O(1) and stable under concurrent writes. Always use keyset for large datasets.

kotlin

// Keyset (cursor) pagination — O(1) vs OFFSET O(n)
@Repository
class ProductRepository(private val dsl: DSLContext) {

    // WRONG: offset pagination — scans and discards rows
    fun findAllOffset(page: Int, size: Int): List<Product> =
        dsl.selectFrom(PRODUCTS)
            .orderBy(PRODUCTS.CREATED_AT.desc())
            .offset(page * size)  // scans ALL previous rows!
            .limit(size)
            .fetchInto(Product::class.java)

    // CORRECT: keyset pagination — seeks directly to the cursor
    fun findAllKeyset(cursor: Instant?, size: Int): List<Product> =
        dsl.selectFrom(PRODUCTS)
            .apply { cursor?.let { where(PRODUCTS.CREATED_AT.lt(it)) } }
            .orderBy(PRODUCTS.CREATED_AT.desc())
            .limit(size + 1) // fetch one extra to know if there's a next page
            .fetchInto(Product::class.java)
}

// Usage: GET /api/v1/products?cursor=2026-01-15T10:30:00Z&size=20
// Response includes: { data: [...], nextCursor: "2026-01-15T09:00:00Z" }

Performance Trap

OFFSET 100000 on a table with millions of rows will scan and discard 100,000 rows before returning results. With keyset pagination, the query jumps directly to the cursor position using an index seek. The difference can be 100x at scale.

Chapter 11: Security & Authentication

Expert

Security is not a feature — it's a constraint that affects every layer. Senior engineers must understand Spring Security configuration, JWT token lifecycle, OAuth2 resource servers, CORS policies, and method-level authorization. A single misconfiguration can expose your entire system.

11.1 Spring Security Kotlin DSL

Spring Security 6+ provides a Kotlin DSL that replaces the verbose Java HttpSecurity builder chain. Configure CSRF (disable for stateless APIs), CORS origins, authorization rules, OAuth2 resource server, and session management in a single readable block.

kotlin

// Spring Security — Kotlin DSL configuration
@Configuration
@EnableWebSecurity
class SecurityConfig {

    @Bean
    fun securityFilterChain(http: HttpSecurity): SecurityFilterChain =
        http {
            csrf { disable() }  // stateless API, use CORS instead
            cors { configurationSource = corsConfig() }

            authorizeHttpRequests {
                authorize("/api/v1/auth/**", permitAll)
                authorize("/api/v1/public/**", permitAll)
                authorize("/actuator/health", permitAll)
                authorize("/api/v1/admin/**", hasRole("ADMIN"))
                authorize(anyRequest, authenticated)
            }

            oauth2ResourceServer {
                jwt {
                    jwtDecoder = jwtDecoder()
                }
            }

            sessionManagement {
                sessionCreationPolicy = SessionCreationPolicy.STATELESS
            }

            exceptionHandling {
                authenticationEntryPoint = HttpStatusEntryPoint(HttpStatus.UNAUTHORIZED)
            }
        }.build()

    @Bean
    fun corsConfig() = CorsConfigurationSource {
        CorsConfiguration().apply {
            allowedOrigins = listOf("https://app.mycompany.com")
            allowedMethods = listOf("GET", "POST", "PUT", "PATCH", "DELETE")
            allowedHeaders = listOf("*")
            allowCredentials = true
        }
    }
}

Interview Deep Dive

Q: Why disable CSRF for REST APIs?
A: CSRF protects against cross-site form submissions that use cookies for authentication. Stateless APIs that use Authorization: Bearer headers are not vulnerable to CSRF because browsers don't automatically attach custom headers. Disabling CSRF for cookie-based auth would be a critical vulnerability.

11.2 JWT Authentication

JWTs enable stateless authentication: the token contains the user's identity and roles, signed with a secret key. No session storage needed. Use HMAC-SHA256 for symmetric signing or RS256 for asymmetric (when multiple services need to validate tokens).

kotlin

// JWT token generation and validation
@Service
class JwtService(private val props: JwtProperties) {

    private val secretKey = Keys.hmacShaKeyFor(props.secret.toByteArray())

    fun generateToken(user: UserPrincipal): String =
        Jwts.builder()
            .subject(user.id.toString())
            .claim("email", user.email)
            .claim("roles", user.roles)
            .issuedAt(Date())
            .expiration(Date(System.currentTimeMillis() + props.expiration.toMillis()))
            .signWith(secretKey)
            .compact()

    fun validateToken(token: String): Claims =
        Jwts.parser()
            .verifyWith(secretKey)
            .build()
            .parseSignedClaims(token)
            .payload

    fun extractUserId(token: String): Long =
        validateToken(token).subject.toLong()
}

// Authentication filter
@Component
class JwtAuthFilter(
    private val jwtService: JwtService,
    private val userService: UserService
) : OncePerRequestFilter() {

    override fun doFilterInternal(
        request: HttpServletRequest,
        response: HttpServletResponse,
        chain: FilterChain
    ) {
        val token = request.getHeader("Authorization")
            ?.removePrefix("Bearer ")
            ?.takeIf { it.isNotBlank() }

        if (token != null) {
            try {
                val userId = jwtService.extractUserId(token)
                val user = userService.loadById(userId)
                val auth = UsernamePasswordAuthenticationToken(user, null, user.authorities)
                SecurityContextHolder.getContext().authentication = auth
            } catch (e: JwtException) {
                // Invalid token — continue without authentication
            }
        }
        chain.doFilter(request, response)
    }
}

Security Gotcha

Never store JWTs in localStorage — vulnerable to XSS. Use HttpOnly cookies for web apps, or keep tokens in memory and use refresh tokens. Always set short expiration times (15-30 min) and implement token rotation. Never put sensitive data in the JWT payload — it's base64-encoded, not encrypted.

11.3 Method-Level Security

Use @PreAuthorize and @PostAuthorize for fine-grained access control at the service layer. Spring Expression Language (SpEL) enables dynamic rules: "only the owner can access this resource" or "admins can access everything". Create custom annotations for reusable security patterns.

kotlin

// Method-level security with custom annotations
@EnableMethodSecurity(prePostEnabled = true)
@Configuration
class MethodSecurityConfig

// Built-in annotations
@Service
class OrderService(private val repo: OrderRepository) {

    @PreAuthorize("hasRole('ADMIN') or #userId == authentication.principal.id")
    suspend fun getOrdersForUser(userId: Long): List<Order> =
        repo.findByUserId(userId)

    @PreAuthorize("hasRole('ADMIN')")
    suspend fun deleteOrder(orderId: Long) =
        repo.deleteById(orderId)

    @PostAuthorize("returnObject.userId == authentication.principal.id")
    suspend fun getOrder(orderId: Long): Order =
        repo.findById(orderId) ?: throw OrderNotFoundException(orderId)
}

// Custom security expression
@Target(AnnotationTarget.FUNCTION)
@Retention(AnnotationRetention.RUNTIME)
@PreAuthorize("hasRole('ADMIN') or @ownershipChecker.isOwner(#id, authentication)")
annotation class OwnerOrAdmin

@Service
class OwnershipChecker(private val repo: OrderRepository) {
    fun isOwner(id: Long, auth: Authentication): Boolean {
        val principal = auth.principal as UserPrincipal
        return repo.findById(id)?.userId == principal.id
    }
}

*Defense in depth: Don't rely on a single layer. Combine: (1) SecurityFilterChain for URL-level rules, (2) @PreAuthorize for method-level, (3) domain-level checks in your business logic. If the security annotation is removed by accident, the domain validation still prevents unauthorized access.

Chapter 12: Performance & Observability

Expert

You can't improve what you can't measure. Staff engineers are expected to design systems with observability built in: multi-level caching, custom business metrics, structured logging with correlation IDs, and Kubernetes-native health checks. This chapter covers the full observability stack.

12.1 Multi-Level Caching

Use a two-tier cache: L1 (Caffeine, in-process, microsecond reads) for hot data, L2 (Redis, shared, millisecond reads) for consistency across instances. Spring's @Cacheable / @CacheEvict annotations make it declarative. Use CompositeCacheManager to chain both.

kotlin

// Multi-level caching — in-memory (Caffeine) + distributed (Redis)
@Configuration
@EnableCaching
class CacheConfig {

    @Bean
    fun cacheManager(redisConnectionFactory: RedisConnectionFactory): CacheManager =
        CompositeCacheManager(
            caffeineCacheManager(),      // L1: in-process, fast
            redisCacheManager(redisConnectionFactory)  // L2: shared, consistent
        ).apply { setFallbackToNoOpCache(false) }

    private fun caffeineCacheManager() = CaffeineCacheManager().apply {
        setCaffeine(Caffeine.newBuilder()
            .maximumSize(1000)
            .expireAfterWrite(Duration.ofMinutes(5))
            .recordStats())
    }

    private fun redisCacheManager(factory: RedisConnectionFactory) =
        RedisCacheManager.builder(factory)
            .cacheDefaults(RedisCacheConfiguration.defaultCacheConfig()
                .entryTtl(Duration.ofMinutes(30))
                .serializeValuesWith(
                    RedisSerializationContext.SerializationPair
                        .fromSerializer(GenericJackson2JsonRedisSerializer())
                ))
            .build()
}

// Usage with cache annotations
@Service
class ProductService(private val repo: ProductRepository) {

    @Cacheable("products", key = "#id")
    suspend fun findById(id: Long): Product =
        repo.findById(id) ?: throw ProductNotFoundException(id)

    @CacheEvict("products", key = "#id")
    suspend fun update(id: Long, cmd: UpdateProductCommand): Product {
        val product = findById(id)
        return repo.save(product.update(cmd))
    }

    @CacheEvict("products", allEntries = true)
    suspend fun bulkImport(products: List<CreateProductCommand>) =
        repo.saveAll(products.map { it.toEntity() })
}

Interview Deep Dive

Q: How do you handle cache invalidation in a multi-instance deployment?
A: L1 (in-memory) caches are instance-local — updates on one instance won't invalidate others. Solutions: (1) Short TTL for L1 (5 min max). (2) Use Redis Pub/Sub to broadcast invalidation events. (3) @CacheEvict only evicts L2 (Redis); L1 expires naturally. (4) For critical data, skip L1 entirely and use only L2.

12.2 Custom Business Metrics

Technical metrics (http_requests_total) are not enough. Track business metrics: orders per minute, revenue by channel, conversion rates. Use Micrometer's Counter, Timer, and Gauge to expose these to Prometheus/Grafana. Add structured logging with correlation IDs for distributed tracing.

kotlin

// Custom business metrics with Micrometer
@Component
class OrderMetrics(private val registry: MeterRegistry) {

    private val orderCounter = registry.counter("orders.created", "type", "all")
    private val orderTimer = registry.timer("orders.processing.time")
    private val activeOrders = registry.gauge("orders.active.count",
        AtomicInteger(0))!!

    fun recordOrderCreated(order: Order) {
        orderCounter.increment()
        registry.counter("orders.created", "channel", order.channel.name).increment()
        registry.counter("orders.revenue",
            "currency", order.currency
        ).increment(order.total.toDouble())
    }

    suspend fun <T> timeOrderProcessing(block: suspend () -> T): T {
        val sample = Timer.start(registry)
        return try {
            block()
        } finally {
            sample.stop(orderTimer)
        }
    }
}

// Structured logging with correlation IDs
@Component
class RequestLoggingFilter : WebFilter {
    override fun filter(exchange: ServerWebExchange, chain: WebFilterChain): Mono<Void> {
        val requestId = exchange.request.headers.getFirst("X-Request-ID")
            ?: UUID.randomUUID().toString()

        return chain.filter(exchange)
            .contextWrite { ctx ->
                ctx.put("requestId", requestId)
            }
            .doOnEach { signal ->
                if (!signal.isOnComplete) {
                    MDC.put("requestId", requestId)
                    MDC.put("method", exchange.request.method.name())
                    MDC.put("path", exchange.request.path.value())
                }
            }
    }
}

12.3 Health Checks & Kubernetes Probes

Kubernetes needs three probes: liveness (is the app alive?), readiness (can it serve traffic?), and startup (has it finished initializing?). Spring Boot Actuator provides these out of the box. Add custom HealthIndicator implementations for each dependency (database, Kafka, external APIs).

kotlin

// Custom health indicators for Kubernetes probes
@Component
class DatabaseHealthIndicator(
    private val dataSource: DataSource
) : HealthIndicator {

    override fun health(): Health {
        return try {
            dataSource.connection.use { conn ->
                conn.createStatement().use { stmt ->
                    stmt.executeQuery("SELECT 1").use { rs ->
                        if (rs.next()) Health.up()
                            .withDetail("database", "PostgreSQL")
                            .withDetail("latency", measureTimeMillis { })
                            .build()
                        else Health.down().build()
                    }
                }
            }
        } catch (e: Exception) {
            Health.down(e).build()
        }
    }
}

@Component
class KafkaHealthIndicator(
    private val adminClient: AdminClient
) : HealthIndicator {

    override fun health(): Health {
        return try {
            val nodes = adminClient.describeCluster().nodes().get(5, TimeUnit.SECONDS)
            Health.up()
                .withDetail("brokers", nodes.size)
                .withDetail("clusterId", adminClient.describeCluster().clusterId().get())
                .build()
        } catch (e: Exception) {
            Health.down(e).withDetail("error", e.message).build()
        }
    }
}

// application.yml
// management:
//   endpoints:
//     web:
//       exposure:
//         include: health, metrics, prometheus
//   endpoint:
//     health:
//       show-details: when_authorized
//       probes:
//         enabled: true  # /actuator/health/liveness, /readiness

*Probe design principle: Liveness should only check if the JVM is healthy — never include external dependency checks (they'd cause unnecessary restarts). Readiness should check database, cache, and message queue connectivity. If readiness fails, Kubernetes removes the pod from the load balancer but doesn't restart it.

Chapter 13: Kotlin Advanced Type System

Advanced

Kotlin's type system goes far beyond null safety. sealed classes enable exhaustive pattern matching, value classes provide type-safe wrappers with zero overhead, delegation replaces inheritance, and DSL builders create domain-specific languages. Mastering these puts you in the top tier of Kotlin developers.

13.1 Sealed Classes & Interfaces

sealed class restricts hierarchy — all subtypes must be defined in the same package. The compiler knows every possible subtype, enabling exhaustive when expressions without else. Adding a new subtype forces you to update every when — the compiler catches missing cases.

kotlin

// Sealed classes — exhaustive when, type-safe hierarchies
sealed class PaymentResult {
    data class Success(val transactionId: String, val amount: Money) : PaymentResult()
    data class Declined(val reason: String, val code: Int) : PaymentResult()
    data class RequiresAction(val actionUrl: String) : PaymentResult()
    data object NetworkError : PaymentResult()
}

// Compiler enforces exhaustive when — no "else" needed
fun handlePayment(result: PaymentResult): String = when (result) {
    is PaymentResult.Success -> "Paid: ${result.transactionId}"
    is PaymentResult.Declined -> "Declined: ${result.reason} (code ${result.code})"
    is PaymentResult.RequiresAction -> "Redirect to: ${result.actionUrl}"
    is PaymentResult.NetworkError -> "Network error, retry later"
    // Adding a new subclass forces updating ALL when expressions
}

// Sealed interfaces — multiple inheritance for domain events
sealed interface DomainEvent {
    val occurredAt: Instant
}

sealed interface OrderEvent : DomainEvent {
    val orderId: OrderId
}

data class OrderPlaced(
    override val orderId: OrderId,
    override val occurredAt: Instant,
    val items: List<OrderItem>
) : OrderEvent

data class OrderShipped(
    override val orderId: OrderId,
    override val occurredAt: Instant,
    val trackingNumber: String
) : OrderEvent

Interview Deep Dive

Q: When would you use sealed class vs enum class?
A: Use enum when all variants are singletons with the same data (e.g., Status.ACTIVE, Status.INACTIVE). Use sealed class when variants carry different data (e.g., Success(data) vs Error(message, code)). Sealed classes are Kotlin's equivalent of Rust's enum or Haskell's algebraic data types.

13.2 Value Classes (Inline Classes)

@JvmInline value class wraps a single property with zero runtime overhead — the wrapper is erased at compile time. Use them to prevent "primitive obsession": instead of passing Long for every ID, use UserId, OrderId, ProductId. The compiler prevents mixing them up.

kotlin

// Value classes — type safety without runtime overhead
@JvmInline
value class UserId(val value: Long)

@JvmInline
value class OrderId(val value: Long)

@JvmInline
value class Email(val value: String) {
    init {
        require(value.contains("@")) { "Invalid email: $value" }
    }
}

@JvmInline
value class Money(val cents: Long) {
    operator fun plus(other: Money) = Money(cents + other.cents)
    operator fun times(quantity: Int) = Money(cents * quantity)

    fun toFormattedString(): String =
        "$${cents / 100}.${(cents % 100).toString().padStart(2, '0')}"

    companion object {
        fun fromDollars(dollars: Double) = Money((dollars * 100).toLong())
    }
}

// Compile-time safety — can't accidentally swap IDs
fun transferMoney(
    from: UserId,      // can't pass an OrderId here!
    to: UserId,
    amount: Money
) { /* ... */ }

// At runtime, these are just Long/String — zero overhead
val userId = UserId(42)
val email = Email("alice@example.com")
val price = Money.fromDollars(29.99)

Common Mistake

Without value classes, this compiles: transferMoney(orderId, userId, amount) — silently swapping user and order IDs. With value classes, it's a compile-time error. This is especially critical in financial systems where ID confusion can cause real money to go to the wrong account.

13.3 Delegation Pattern

Kotlin's by keyword enables class delegation: implement an interface by delegating to another object. Override only the methods you need — the rest forward automatically. This replaces inheritance-heavy designs with clean composition. Property delegation (lazy, observable, map) is equally powerful.

kotlin

// Delegation — composition over inheritance
interface Logger {
    fun info(message: String)
    fun error(message: String, throwable: Throwable? = null)
}

class Slf4jLogger(name: String) : Logger {
    private val delegate = LoggerFactory.getLogger(name)
    override fun info(message: String) = delegate.info(message)
    override fun error(message: String, throwable: Throwable?) =
        delegate.error(message, throwable)
}

// Class delegation with "by" keyword
class AuditedOrderService(
    private val delegate: OrderService,
    private val logger: Logger
) : OrderService by delegate {
    // Only override what you need — rest delegates automatically
    override suspend fun create(command: CreateOrderCommand): Order {
        logger.info("Creating order for customer: ${command.customerId}")
        return delegate.create(command).also {
            logger.info("Order created: ${it.id}")
        }
    }
}

// Property delegation — lazy, observable, map-backed
class AppConfig(properties: Map<String, String>) {
    val dbUrl: String by properties           // backed by map
    val maxRetries: Int by lazy { loadFromVault("max-retries").toInt() }
    var debugMode: Boolean by Delegates.observable(false) { _, old, new ->
        logger.info("Debug mode changed: $old -> $new")
    }
}

13.4 Type-Safe DSL Builders

Kotlin's lambda-with-receiver (T.() -> Unit) enables building fluent DSLs. This is the same mechanism behind buildList, apply, Ktor routes, Gradle build scripts, and Spring's Kotlin DSL. Understanding this pattern lets you create domain-specific APIs that read like English.

kotlin

// Type-safe DSL builders — Kotlin's killer feature
// HTML DSL example
fun html(init: HtmlBuilder.() -> Unit): String =
    HtmlBuilder().apply(init).build()

class HtmlBuilder {
    private val elements = mutableListOf<String>()

    fun head(init: HeadBuilder.() -> Unit) {
        elements += HeadBuilder().apply(init).build()
    }

    fun body(init: BodyBuilder.() -> Unit) {
        elements += BodyBuilder().apply(init).build()
    }

    fun build() = "<html>${elements.joinToString("\n")}</html>"
}

// Query DSL for your domain
class QueryBuilder<T> {
    private val conditions = mutableListOf<String>()
    private var orderField: String? = null
    private var limitValue: Int = 100

    fun where(field: String, op: String, value: Any) {
        conditions += "$field $op '$value'"
    }

    fun orderBy(field: String) { orderField = field }
    fun limit(n: Int) { limitValue = n }

    infix fun String.eq(value: Any) = where(this, "=", value)
    infix fun String.gt(value: Any) = where(this, ">", value)
}

// Usage — reads like English
val query = buildQuery<User> {
    "status" eq "active"
    "age" gt 18
    orderBy("created_at")
    limit(50)
}

*When to create a DSL: When you find yourself building complex configuration objects with many optional parameters. If the builder pattern would require 10+ .withX() calls, a DSL with lambda-with-receiver will be more readable and discoverable via IDE autocomplete.

Chapter 14: DevOps & Deployment

Advanced

Staff engineers own the full lifecycle — from code to production. This chapter covers multi-stage Docker builds, GraalVM native compilation for instant startup, Kubernetes deployments with proper health probes, and Helm charts for parameterized deployments. These are the skills that turn a backend developer into a platform engineer.

14.1 Multi-Stage Docker Builds

Always use multi-stage builds: the builder stage has the JDK and Gradle for compilation, the runtime stage has only the JRE. This reduces image size from ~800MB to ~200MB. Run as a non-root user, tune JVM flags for containers (-XX:+UseContainerSupport), and add a HEALTHCHECK instruction.

dockerfile

# Multi-stage Dockerfile for Kotlin Spring Boot
FROM eclipse-temurin:21-jdk-alpine AS builder
WORKDIR /app

# Cache Gradle dependencies
COPY build.gradle.kts settings.gradle.kts gradle.properties ./
COPY gradle ./gradle
RUN ./gradlew dependencies --no-daemon

# Build the app
COPY src ./src
RUN ./gradlew bootJar --no-daemon -x test

# Runtime stage — minimal image
FROM eclipse-temurin:21-jre-alpine
WORKDIR /app

# Security: non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

COPY --from=builder /app/build/libs/*.jar app.jar

# JVM tuning for containers
ENV JAVA_OPTS="-XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -XX:+UseContainerSupport \
  -XX:MaxRAMPercentage=75.0 \
  -Djava.security.egd=file:/dev/./urandom"

EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=3s \
  CMD wget -q --spider http://localhost:8080/actuator/health || exit 1

ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"]

Interview Deep Dive

Q: Why is -XX:MaxRAMPercentage=75.0 important in containers?
A: Without it, the JVM may try to use more memory than the container limit, causing OOMKill. UseContainerSupport makes the JVM aware of cgroup limits, and MaxRAMPercentage caps heap usage at 75% of the container memory, leaving room for off-heap memory (metaspace, thread stacks, NIO buffers).

14.2 GraalVM Native Images

GraalVM compiles your Kotlin/Spring Boot app to a native binary at build time. Result: ~50ms startup (vs ~3s JVM), ~50MB memory (vs ~256MB), 25MB binary. The tradeoff: longer build times (~5 min), no JIT optimization at runtime, and reflection must be declared ahead of time.

kotlin

// GraalVM Native Image — millisecond startup
// build.gradle.kts
plugins {
    id("org.graalvm.buildtools.native") version "0.10.4"
}

graalvmNative {
    binaries {
        named("main") {
            imageName.set("my-service")
            mainClass.set("com.myapp.ApplicationKt")
            buildArgs.add("--enable-preview")
            buildArgs.add("-H:+ReportExceptionStackTraces")
            // Spring AOT processes beans at build time
            jvmArgs.add("-Dspring.aot.enabled=true")
        }
    }
}

// Reflection hints for GraalVM (if needed)
@RegisterReflectionForBinding(
    UserDto::class,
    OrderDto::class,
    PaymentResult::class
)
@SpringBootApplication
class Application

// Build: ./gradlew nativeCompile
// Result: 25MB binary, starts in ~50ms vs ~3s JVM
// Tradeoff: longer build time (~5min), no JIT optimization at runtime

Metric	JVM (JIT)	GraalVM Native
Startup time	2-5 seconds	~50 milliseconds
Memory usage	256-512 MB	50-100 MB
Peak throughput	Higher (JIT optimized)	Lower (AOT only)
Build time	~30 seconds	~5 minutes
Best for	Long-running services	Serverless, CLI, scale-to-zero

14.3 Kubernetes Deployment

A production Kubernetes deployment needs: RollingUpdate strategy with maxUnavailable: 0 for zero-downtime, proper resource requests/limits, liveness/readiness/startup probes, Prometheus annotations for metrics scraping, and secrets mounted from SecretKeyRef.

yaml

# Kubernetes Deployment with proper health checks
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  labels:
    app: order-service
    version: v1
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0    # zero-downtime deployments
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/actuator/prometheus"
    spec:
      containers:
        - name: order-service
          image: registry.mycompany.com/order-service:1.2.3
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "1000m"
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: "prod"
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: password
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
          startupProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            failureThreshold: 30
            periodSeconds: 2

14.4 Helm Charts for Parameterized Deployments

Helm packages Kubernetes manifests as reusable charts with environment-specific values. Use values.yaml for defaults, override with values-prod.yaml. Enable HorizontalPodAutoscaler for auto-scaling, configure Ingress with TLS via cert-manager, and set rate limits at the ingress level.

yaml

# Helm values.yaml — parameterized deployment
replicaCount: 3

image:
  repository: registry.mycompany.com/order-service
  tag: "1.2.3"
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 8080

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/rate-limit: "100"
  hosts:
    - host: api.mycompany.com
      paths:
        - path: /api/v1/orders
          pathType: Prefix
  tls:
    - secretName: api-tls
      hosts:
        - api.mycompany.com

resources:
  requests:
    memory: 256Mi
    cpu: 250m
  limits:
    memory: 512Mi
    cpu: 1000m

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80

# Environment-specific overrides
# helm install order-service ./chart -f values-prod.yaml

*Production checklist: (1) Non-root container user. (2) Resource limits set. (3) All three probes configured. (4) Secrets never in environment variables — use mounted secrets or external secret managers (Vault, AWS Secrets Manager). (5) HPA enabled with CPU and memory targets. (6) PodDisruptionBudget for maintenance windows.

Chapter 15: Microservices Architecture

Expert

Microservices decompose a monolith into independently deployable services, each owning its own data store and business capability. While the pattern enables team autonomy and independent scaling, it introduces significant distributed systems complexity — network partitions, data consistency challenges, and operational overhead. This chapter covers the gateway pattern, service discovery, and inter-service communication strategies that production systems at scale actually use.

What you will learn

API Gateway pattern with Spring Cloud Gateway — routing, rate limiting, and circuit breakers
Service discovery with Kubernetes-native approaches and load-balanced WebClient
Event-driven inter-service communication with Kafka for eventual consistency
When to choose REST, gRPC, or async messaging between services

15.1 API Gateway Pattern

The API Gateway is the single entry point for all client traffic. It handles cross-cutting concerns like authentication, rate limiting, circuit breaking, and request routing — keeping these concerns out of individual microservices. Spring Cloud Gateway provides a reactive, non-blocking gateway built on Project Reactor and Netty. Each route can have its own CircuitBreaker, retry policy, and RequestRateLimiter. Always remove sensitive headers (like Cookie) before forwarding to internal services, and use lb:// URIs for load-balanced routing via service discovery.

kotlin

// Spring Cloud Gateway — Kotlin DSL routes
@Configuration
class GatewayConfig {

    @Bean
    fun routeLocator(builder: RouteLocatorBuilder): RouteLocator =
        builder.routes {
            route("order-service") {
                path("/api/v1/orders/**")
                filters {
                    circuitBreaker {
                        setName("orderServiceCB")
                        setFallbackUri("forward:/fallback/orders")
                    }
                    retry { setRetries(3) }
                    requestRateLimiter {
                        setRateLimiter(redisRateLimiter())
                        setKeyResolver(userKeyResolver())
                    }
                    removeRequestHeader("Cookie") // no cookies to microservices
                }
                uri("lb://order-service") // load-balanced via service discovery
            }

            route("user-service") {
                path("/api/v1/users/**")
                filters {
                    circuitBreaker { setName("userServiceCB") }
                    addRequestHeader("X-Internal-Call", "true")
                }
                uri("lb://user-service")
            }
        }
}

Client→API Gateway→Order Service→Kafka Events

15.2 Service Discovery & Internal Communication

In a microservices world, services must find each other without hardcoded URLs. Kubernetes-native discovery uses DNS (http://order-service resolves via kube-dns), while Spring Cloud Discovery adds features like health-aware routing and zone affinity. For internal high-throughput communication, gRPC offers binary serialization (Protocol Buffers) with ~10x lower latency than JSON REST. Use @LoadBalanced WebClient for HTTP-based calls and gRPC for performance-critical service-to-service paths.

kotlin

// Service registration with Spring Cloud + Kubernetes
// application.yml — each microservice registers itself
// spring:
//   application:
//     name: order-service
//   cloud:
//     kubernetes:
//       discovery:
//         enabled: true
//         all-namespaces: false

// Using WebClient with load balancing (no hardcoded URLs)
@Service
class UserClient(
    @LoadBalanced private val webClientBuilder: WebClient.Builder
) {
    private val client = webClientBuilder
        .baseUrl("http://user-service") // resolved via service discovery
        .build()

    suspend fun getUser(userId: Long): UserDto =
        client.get()
            .uri("/api/v1/users/{id}", userId)
            .retrieve()
            .awaitBody<UserDto>()
}

// gRPC for internal high-performance communication
@GrpcService
class OrderGrpcService(
    private val orderService: OrderService
) : OrderServiceGrpcKt.OrderServiceCoroutineImplBase() {

    override suspend fun getOrder(request: GetOrderRequest): OrderResponse {
        val order = orderService.findById(OrderId(request.orderId))
        return order.toGrpcResponse()
    }
}

Interview Deep Dive

Q: When would you choose gRPC over REST for inter-service communication?
A: gRPC excels for high-throughput, low-latency internal communication: binary serialization is ~5-10x faster than JSON, HTTP/2 multiplexing avoids head-of-line blocking, and .proto files provide strong contracts. REST is better for external APIs (browser-friendly, widely tooled) and services with simple request/response patterns. A common production setup uses REST at the gateway for external clients and gRPC internally between services.

15.3 Event-Driven Microservice Communication

Synchronous inter-service calls create tight coupling and cascade failures. Event-driven architecture decouples services: the Order Service publishes an OrderCreatedEvent to Kafka, and the Inventory Service consumes it asynchronously. This achieves eventual consistency — the inventory won't update in the same millisecond, but the system is resilient to individual service outages. The key pattern is "publish events for state changes, consume events for reactions."

kotlin

// Event-driven communication between microservices
// Order Service publishes events to Kafka
@Service
class OrderEventPublisher(
    private val kafka: KafkaTemplate<String, DomainEvent>
) {
    suspend fun publishOrderCreated(order: Order) {
        val event = OrderCreatedEvent(
            orderId = order.id.value,
            customerId = order.customerId.value,
            totalAmount = order.total.cents,
            items = order.items.map { it.toEventItem() },
            timestamp = Instant.now()
        )
        kafka.send("order-events", order.id.toString(), event).await()
    }
}

// Inventory Service consumes events — eventually consistent
@Component
class InventoryEventConsumer(
    private val inventoryService: InventoryService
) {
    @KafkaListener(
        topics = ["order-events"],
        groupId = "inventory-service",
        containerFactory = "kafkaListenerContainerFactory"
    )
    suspend fun handleOrderEvent(event: OrderCreatedEvent) {
        event.items.forEach { item ->
            inventoryService.reserveStock(
                productId = ProductId(item.productId),
                quantity = item.quantity,
                orderId = OrderId(event.orderId)
            )
        }
    }
}

Real-World Scenario

E-commerce order flow: When a customer places an order, the Order Service saves it and publishes OrderCreatedEvent. The Inventory Service reserves stock, the Payment Service charges the card, and the Notification Service sends a confirmation email — all reacting independently to the same event. If the Payment Service is temporarily down, messages queue in Kafka and are processed when it recovers. No data is lost, and other services continue unaffected.

Key Takeaway

Start with a modular monolith and extract microservices only when you have a clear organizational or scaling reason. Premature decomposition is the #1 microservices anti-pattern — you'll spend more time on infrastructure than on business logic.

Chapter 16: Database Design & Optimization

Advanced

Database performance is the bottleneck of most backend systems. A well-indexed PostgreSQL database can serve 50,000 queries per second on modest hardware, while a poorly indexed one struggles at 500. This chapter covers strategic indexing (the single highest-impact optimization), query patterns to avoid (N+1 is the classic trap), and connection pool tuning that prevents your application from starving under load.

What you will learn

Composite, partial, covering, and expression indexes — when to use each type
Detecting and eliminating N+1 queries with JOINs and batch fetching
HikariCP connection pool sizing and tuning for production workloads
Read replica routing for scaling read-heavy workloads

16.1 Strategic Indexing

Indexes are the single biggest performance lever in any relational database. A composite index on (customer_id, status, created_at DESC) covers your most common query pattern in a single B-tree lookup. Partial indexes (with a WHERE clause) index only the rows you actually query — a partial index on status = 'PENDING' is 90% smaller than a full index. Covering indexes use INCLUDE to add non-key columns, enabling index-only scans that skip the table heap entirely. Expression indexes like LOWER(email) support case-insensitive lookups efficiently.

sql

// Strategic indexing — the single biggest performance lever
-- Composite index for common query patterns
CREATE INDEX idx_orders_customer_status_created
    ON orders (customer_id, status, created_at DESC);
-- Covers: WHERE customer_id = ? AND status = ? ORDER BY created_at DESC

-- Partial index — only index what you actually query
CREATE INDEX idx_orders_pending
    ON orders (created_at DESC)
    WHERE status = 'PENDING';
-- 90% smaller than a full index, 10x faster for pending order queries

-- Covering index — eliminates table lookups entirely
CREATE INDEX idx_products_category_covering
    ON products (category_id)
    INCLUDE (name, price, stock_count);
-- SELECT name, price, stock_count WHERE category_id = ? → index-only scan

-- Expression index for case-insensitive search
CREATE INDEX idx_users_email_lower
    ON users (LOWER(email));
-- WHERE LOWER(email) = 'alice@example.com' → uses the index

Common Pitfall

Don't blindly add indexes. Each index slows down writes (INSERT/UPDATE must update the index B-tree), consumes disk space, and adds vacuum overhead. For a write-heavy table with 50M rows, a poorly chosen index can make writes 2-3x slower. Always check EXPLAIN ANALYZE to verify the optimizer uses your index before leaving it in production.

16.2 Query Optimization Patterns

The N+1 problem is the most common performance bug in ORM-based applications. It happens when you load a list of entities, then lazily load a related entity for each one — resulting in 1 query for the list plus N queries for the relations. The fix is straightforward: use an explicit JOIN query with projection, or use @EntityGraph in JPA to eager-fetch relations in a single query. This turns N+1 queries into 1 query, often reducing response time from seconds to milliseconds.

kotlin

// Query optimization patterns in Kotlin
@Repository
class OrderAnalyticsRepository(private val jdbcTemplate: NamedParameterJdbcTemplate) {

    // BAD: N+1 — loads orders, then each customer separately
    suspend fun getOrdersWithCustomersBad(): List<OrderWithCustomer> {
        val orders = orderRepo.findAll() // query 1
        return orders.map { order ->
            val customer = customerRepo.findById(order.customerId) // query N
            OrderWithCustomer(order, customer!!)
        }
    }

    // GOOD: Single query with JOIN + projection
    fun getOrdersWithCustomersGood(
        status: OrderStatus,
        limit: Int
    ): List<OrderWithCustomer> =
        jdbcTemplate.query("""
            SELECT o.id, o.total, o.created_at,
                   c.id AS customer_id, c.name, c.email
            FROM orders o
            JOIN customers c ON c.id = o.customer_id
            WHERE o.status = :status
            ORDER BY o.created_at DESC
            LIMIT :limit
        """, mapOf("status" to status.name, "limit" to limit)) { rs, _ ->
            OrderWithCustomer(
                orderId = rs.getLong("id"),
                total = rs.getBigDecimal("total"),
                customerName = rs.getString("name"),
                customerEmail = rs.getString("email")
            )
        }
}

Interview Deep Dive

Q: You have a query that takes 3 seconds. Walk me through your debugging process.
A: (1) Run EXPLAIN ANALYZE to get the execution plan — look for sequential scans on large tables, nested loop joins, and sort operations. (2) Check if the right indexes exist for the WHERE/ORDER BY columns. (3) Verify the query planner is using the indexes (sometimes a stale statistics leads to bad plans — run ANALYZE). (4) Consider if the query can be restructured to use a covering index. (5) If it's a reporting query, consider materializing it or using a read replica.

16.3 Connection Pool Tuning

Connection pools prevent your application from opening/closing database connections on every request (each connection costs ~3MB of PostgreSQL memory and ~150ms to establish). HikariCP is the default in Spring Boot and the fastest JVM connection pool. The optimal pool size follows the formula: (2 * CPU cores) + number of spinning disks. For a 4-core server with SSDs, that's about 10 connections. More connections means more contention and context switching — bigger is not better. Set maxLifetime slightly below the database's connection timeout to prevent stale connections, and enable leakDetectionThreshold in development to catch leaked connections early.

kotlin

// HikariCP tuning for production workloads
@Configuration
class DataSourceConfig {

    @Bean
    @ConfigurationProperties(prefix = "spring.datasource.hikari")
    fun dataSource(): HikariDataSource = HikariDataSource().apply {
        // Pool size = (2 * CPU cores) + number of spinning disks
        // For SSD with 4 cores: pool size = 10
        maximumPoolSize = 10
        minimumIdle = 5

        // Connection lifetime — rotate before DB timeout (usually 30 min)
        maxLifetime = Duration.ofMinutes(25).toMillis()

        // Fail fast if no connection available
        connectionTimeout = Duration.ofSeconds(5).toMillis()

        // Detect leaked connections in development
        leakDetectionThreshold = Duration.ofSeconds(30).toMillis()

        // Validation query (lightweight, no table scan)
        connectionTestQuery = "SELECT 1"
    }
}

// application.yml for read replicas
// spring:
//   datasource:
//     primary:
//       url: jdbc:postgresql://primary-db:5432/myapp
//     replica:
//       url: jdbc:postgresql://replica-db:5432/myapp
//       hikari:
//         read-only: true

Key Takeaway

Most database performance problems are solved by three things: proper indexes, eliminating N+1 queries, and right-sizing the connection pool. Master these before reaching for caching, sharding, or read replicas. A well-indexed single PostgreSQL instance can handle remarkably high throughput.

Chapter 17: Event-Driven Architecture

Expert

Event-driven architecture (EDA) is the backbone of modern distributed systems. Instead of synchronous request-response chains, services communicate through events — immutable facts that describe something that happened. This decouples producers from consumers, enables temporal decoupling (consumers can be offline), and creates a natural audit trail. This chapter covers Apache Kafka configuration for exactly-once delivery, the Transactional Outbox pattern for atomic database + event publishing, consumer error handling with Dead Letter Queues, and Event Sourcing for reconstructing state from event history.

What you will learn

Kafka producer configuration with idempotence and exactly-once semantics
Transactional Outbox pattern — solving the dual-write problem
Consumer error handling with manual acknowledgment and Dead Letter Queues
Event Sourcing — storing events as the source of truth and rebuilding state

17.1 Kafka Producer & Transactional Outbox

The dual-write problem is a classic distributed systems trap: you need to save data to your database AND publish an event to Kafka, but these are two separate systems that can't participate in the same transaction. If the DB write succeeds but Kafka publish fails, you have inconsistency. The Transactional Outbox pattern solves this elegantly: write the event to an outbox table in the same database transaction as your business data. A separate poller (or CDC connector like Debezium) reads the outbox and publishes to Kafka. This guarantees at-least-once delivery with no data loss.

kotlin

// Kafka producer with guaranteed delivery
@Configuration
class KafkaProducerConfig {

    @Bean
    fun producerFactory(): ProducerFactory<String, DomainEvent> =
        DefaultKafkaProducerFactory(mapOf(
            ProducerConfig.BOOTSTRAP_SERVERS_CONFIG to "kafka:9092",
            ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG to StringSerializer::class.java,
            ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG to JsonSerializer::class.java,
            // Exactly-once semantics
            ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG to true,
            ProducerConfig.ACKS_CONFIG to "all",
            ProducerConfig.RETRIES_CONFIG to 3,
            ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION to 1
        ))
}

// Transactional outbox pattern — atomic DB + event publishing
@Service
class OrderService(
    private val orderRepo: OrderRepository,
    private val outboxRepo: OutboxRepository,
    @Transactional private val txManager: PlatformTransactionManager
) {
    @Transactional
    suspend fun createOrder(command: CreateOrderCommand): Order {
        val order = Order.create(command.customerId, command.items)
        val saved = orderRepo.save(order)

        // Write event to outbox table in SAME transaction
        outboxRepo.save(OutboxEvent(
            aggregateType = "Order",
            aggregateId = saved.id.toString(),
            eventType = "OrderCreated",
            payload = Json.encodeToString(OrderCreatedEvent(saved))
        ))

        return saved
        // A separate poller reads outbox and publishes to Kafka
    }
}

Interview Deep Dive

Q: Explain the difference between at-most-once, at-least-once, and exactly-once delivery in Kafka.
A: At-most-once: consumer commits offset before processing — if it crashes mid-processing, the message is skipped (data loss). At-least-once: consumer processes first, then commits — if it crashes after processing but before committing, the message is reprocessed (duplicates). Exactly-once: Kafka's idempotent producer (enable.idempotence=true) prevents duplicate writes, and transactional consumers use read-committed isolation to see only completed transactions. In practice, most systems use at-least-once with idempotent consumers (processing the same event twice produces the same result).

17.2 Kafka Consumer & Error Handling

Consumer reliability is just as important as producer reliability. Disable auto.commit and use manual acknowledgment — this ensures you only commit offsets after successful processing. When a message fails, the DefaultErrorHandler retries with configurable backoff (e.g., 3 retries with 1-second intervals). After exhausting retries, the message is sent to a Dead Letter Queue (DLQ) topic for manual investigation. Always set concurrency on the listener container to match your partition count for maximum parallelism.

kotlin

// Kafka consumer with error handling and DLQ
@Configuration
@EnableKafka
class KafkaConsumerConfig {

    @Bean
    fun consumerFactory(): ConsumerFactory<String, String> =
        DefaultKafkaConsumerFactory(mapOf(
            ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG to "kafka:9092",
            ConsumerConfig.GROUP_ID_CONFIG to "payment-service",
            ConsumerConfig.AUTO_OFFSET_RESET_CONFIG to "earliest",
            ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG to false, // manual commit
            ConsumerConfig.MAX_POLL_RECORDS_CONFIG to 100
        ))

    @Bean
    fun kafkaListenerContainerFactory(
        consumerFactory: ConsumerFactory<String, String>
    ): ConcurrentKafkaListenerContainerFactory<String, String> =
        ConcurrentKafkaListenerContainerFactory<String, String>().apply {
            this.consumerFactory = consumerFactory
            setConcurrency(3) // 3 consumer threads
            containerProperties.ackMode = ContainerProperties.AckMode.MANUAL_IMMEDIATE

            // Retry 3 times, then send to DLQ
            setCommonErrorHandler(DefaultErrorHandler(
                DeadLetterPublishingRecoverer(kafkaTemplate),
                FixedBackOff(1000L, 3)
            ))
        }
}

// Consumer with manual acknowledgment
@Component
class PaymentEventConsumer(private val paymentService: PaymentService) {

    @KafkaListener(topics = ["order-events"], groupId = "payment-service")
    suspend fun handleOrderCreated(
        record: ConsumerRecord<String, String>,
        ack: Acknowledgment
    ) {
        val event = Json.decodeFromString<OrderCreatedEvent>(record.value())

        try {
            paymentService.processPayment(event)
            ack.acknowledge() // commit offset only on success
        } catch (e: RetryableException) {
            throw e // will be retried by error handler
        } catch (e: Exception) {
            logger.error("Non-retryable error for order ${event.orderId}", e)
            ack.acknowledge() // skip to DLQ
        }
    }
}

Common Pitfall

Never use auto.commit=true in production. If your consumer crashes after polling but before processing, those messages are lost (committed but not processed). Manual acknowledgment with AckMode.MANUAL_IMMEDIATE ensures you only commit offsets for messages you've actually handled. This is the difference between "works in dev" and "works in production."

17.3 Event Sourcing

Event Sourcing flips the traditional persistence model: instead of storing the current state, you store every state change as an immutable event. The current state is reconstructed by replaying events from the beginning (or from a snapshot). This provides a complete audit trail, enables temporal queries ("what was the order status at 3pm yesterday?"), and allows projecting the same events into different read models. The tradeoff is complexity — you need event versioning, snapshot optimization, and CQRS (Command Query Responsibility Segregation) to keep read performance acceptable as event history grows.

kotlin

// Event Sourcing — reconstruct state from event history
// Events are immutable facts that happened
sealed class OrderEvent {
    abstract val orderId: String
    abstract val timestamp: Instant

    data class OrderPlaced(
        override val orderId: String,
        override val timestamp: Instant,
        val customerId: String,
        val items: List<OrderItem>
    ) : OrderEvent()

    data class PaymentReceived(
        override val orderId: String,
        override val timestamp: Instant,
        val transactionId: String,
        val amount: Money
    ) : OrderEvent()

    data class OrderShipped(
        override val orderId: String,
        override val timestamp: Instant,
        val trackingNumber: String,
        val carrier: String
    ) : OrderEvent()
}

// Rebuild current state by replaying events
class OrderAggregate {
    var status: OrderStatus = OrderStatus.UNKNOWN
        private set
    var items: List<OrderItem> = emptyList()
        private set
    var trackingNumber: String? = null
        private set

    fun apply(event: OrderEvent) {
        when (event) {
            is OrderEvent.OrderPlaced -> {
                status = OrderStatus.PLACED
                items = event.items
            }
            is OrderEvent.PaymentReceived -> {
                status = OrderStatus.PAID
            }
            is OrderEvent.OrderShipped -> {
                status = OrderStatus.SHIPPED
                trackingNumber = event.trackingNumber
            }
        }
    }

    companion object {
        fun fromHistory(events: List<OrderEvent>): OrderAggregate =
            OrderAggregate().apply { events.forEach { apply(it) } }
    }
}

Real-World Scenario

Financial systems: Banks and payment processors use Event Sourcing because regulators require a complete, immutable audit trail of every state change. If a dispute arises, you can replay events to reconstruct exactly what happened and when. Traditional CRUD systems lose this history — once you UPDATE a row, the previous state is gone forever.

Key Takeaway

Event-driven architecture and Event Sourcing are powerful but complex patterns. Use event-driven communication between microservices for loose coupling. Reserve full Event Sourcing for domains where audit trails and temporal queries are genuine requirements (financial, healthcare, compliance). For most CRUD services, traditional persistence with an outbox pattern is simpler and sufficient.

Chapter 18: Kotlin Coroutines Deep Dive

Expert

Chapter 5 introduced coroutines vs virtual threads. This chapter goes deeper into the mechanics: dispatchers (where coroutines execute), structured concurrency (how failures propagate), exception handling patterns for production reliability, and testing strategies with virtual time. Mastering these concepts separates engineers who "use coroutines" from engineers who "understand coroutines." At the staff level, you're expected to design coroutine architectures, debug subtle concurrency bugs, and teach your team the right patterns.

What you will learn

Dispatcher selection — Default for CPU, IO for blocking, limitedParallelism for resource protection
Structured concurrency with supervisorScope for graceful partial failures
CoroutineExceptionHandler and the runCatching pattern for production error handling
Testing coroutines with runTest, StandardTestDispatcher, and Turbine for Flows

18.1 Dispatchers — Where Coroutines Execute

Every coroutine runs on a dispatcher, which determines which thread (or thread pool) executes it. Dispatchers.Default uses a shared pool sized to the number of CPU cores — use it for CPU-bound computation like JSON parsing or analytics. Dispatchers.IO uses a larger elastic pool (up to 64 threads) designed for blocking I/O like JDBC calls or file reads. limitedParallelism(n) creates a view of a dispatcher with at most n concurrent coroutines — perfect for protecting limited resources like database connection pools. The critical rule: never do blocking I/O on Dispatchers.Default — it will starve the shared pool and freeze all CPU-bound work in the application.

kotlin

// Understanding Dispatchers — where coroutines execute
suspend fun processOrder(orderId: Long) = coroutineScope {
    // Dispatchers.Default — CPU-bound work (computation, JSON parsing)
    val analytics = async(Dispatchers.Default) {
        calculateOrderAnalytics(orderId) // heavy computation
    }

    // Dispatchers.IO — blocking I/O (JDBC, file reads, legacy APIs)
    val orderData = async(Dispatchers.IO) {
        legacyOrderDao.findById(orderId) // blocking JDBC call
    }

    // Dispatchers.Unconfined — starts in caller's thread, resumes in
    // whatever thread the suspend function resumes on (rarely used)

    // No dispatcher = inherits from parent scope
    val enrichedOrder = async {
        enrichOrder(orderData.await()) // inherits parent dispatcher
    }

    OrderResult(
        analytics = analytics.await(),
        order = enrichedOrder.await()
    )
}

// Custom limited dispatcher for resource protection
val dbDispatcher = Dispatchers.IO.limitedParallelism(10)
// Max 10 concurrent DB operations — prevents connection pool exhaustion

suspend fun queryWithLimit() = withContext(dbDispatcher) {
    repository.findExpensiveQuery() // at most 10 concurrent calls
}

Interview Deep Dive

Q: What happens if you call a blocking JDBC query on Dispatchers.Default?
A: Dispatchers.Default has only N threads (where N = CPU cores). If all threads are blocked on JDBC calls, no CPU-bound coroutines can execute — the entire application hangs. This is called "thread starvation." The fix: always wrap blocking calls in withContext(Dispatchers.IO), or better, use Dispatchers.IO.limitedParallelism(poolSize) to cap concurrent blocking operations at your connection pool size.

18.2 Structured Concurrency & Exception Handling

Structured concurrency ensures that when a parent coroutine is cancelled, all its children are cancelled too — no orphan coroutines leaking resources. supervisorScope is the key pattern for production code: unlike regular coroutineScope, a child failure in supervisorScope doesn't cancel siblings. This is critical for dashboard-style APIs where you fetch data from multiple sources and want partial results even if one source fails. Combine supervisorScope with runCatching and getOrElse to provide graceful fallbacks for each parallel operation.

kotlin

// Structured exception handling in coroutines
// SupervisorJob — child failures don't cancel siblings
val scope = CoroutineScope(SupervisorJob() + Dispatchers.Default)

// CoroutineExceptionHandler — last resort for uncaught exceptions
val handler = CoroutineExceptionHandler { _, exception ->
    logger.error("Uncaught coroutine exception", exception)
    metrics.incrementCounter("coroutine.unhandled.errors")
}

// Pattern: supervised fan-out with individual error handling
suspend fun fetchDashboardData(userId: Long): Dashboard = supervisorScope {
    // Each async can fail independently
    val profile = async {
        runCatching { userService.getProfile(userId) }
            .getOrElse { UserProfile.DEFAULT }
    }

    val orders = async {
        runCatching { orderService.getRecent(userId) }
            .getOrElse { emptyList() }
    }

    val recommendations = async {
        runCatching { mlService.recommend(userId) }
            .getOrDefault(emptyList())
    }

    Dashboard(
        profile = profile.await(),
        orders = orders.await(),
        recommendations = recommendations.await()
    )
    // Even if ML service is down, dashboard renders with partial data
}

Real-World Scenario

Dashboard API: Your endpoint fetches user profile, recent orders, and ML recommendations in parallel. The ML service has 99.5% uptime — it fails once every 200 requests. With regular coroutineScope, one ML failure cancels everything and returns a 500. With supervisorScope + runCatching, the dashboard renders with the profile and orders, and shows "recommendations unavailable" — a much better user experience.

18.3 Testing Coroutines

Coroutine tests use runTest which provides virtual time — delays complete instantly without actual waiting. Use StandardTestDispatcher to control exactly when coroutines execute (unlike UnconfinedTestDispatcher which runs eagerly). For testing Flow emissions, the Turbine library provides an elegant API: flow.test { awaitItem() } lets you assert on each emission sequentially. Always inject dispatchers rather than hardcoding them, so tests can substitute test dispatchers for predictable, deterministic execution.

kotlin

// Testing coroutines with runTest and turbine
class OrderServiceTest {

    @Test
    fun `should process orders concurrently`() = runTest {
        val service = OrderService(
            repo = FakeOrderRepository(),
            dispatcher = StandardTestDispatcher(testScheduler)
        )

        val result = service.processOrder(OrderId(1))

        assertThat(result.status).isEqualTo(OrderStatus.CONFIRMED)
        // Virtual time — no real delays, tests run instantly
    }

    @Test
    fun `should emit price updates via Flow`() = runTest {
        val priceFlow = priceService.streamPrices("AAPL")

        priceFlow.test { // turbine extension
            val first = awaitItem()
            assertThat(first.symbol).isEqualTo("AAPL")

            val second = awaitItem()
            assertThat(second.price).isGreaterThan(0)

            cancelAndConsumeRemainingEvents()
        }
    }

    @Test
    fun `should cancel children on timeout`() = runTest {
        val result = withTimeoutOrNull(Duration.ofSeconds(5)) {
            slowExternalService.call() // takes 10s
        }

        assertThat(result).isNull() // timed out gracefully
    }
}

Key Takeaway

Coroutines are Kotlin's most powerful concurrency primitive, but power demands discipline. Use the right dispatcher for each workload, prefer supervisorScope for parallel operations that can partially fail, and always inject dispatchers for testability. A well-designed coroutine architecture handles partial failures gracefully and never starves shared thread pools.

Chapter 19: GraphQL with Kotlin

Advanced

GraphQL gives clients the power to request exactly the data they need — no more over-fetching (getting 30 fields when you need 3) and no more under-fetching (making 5 REST calls to assemble one view). The Netflix DGS Framework brings GraphQL to Kotlin/Spring Boot with type-safe resolvers, DataLoaders for batch optimization, and native coroutine support. This chapter covers schema design, resolver implementation, and the critical N+1 problem in GraphQL (which is even more dangerous than in REST because clients control the query shape).

What you will learn

GraphQL schema design with proper types, mutations, and subscriptions
Netflix DGS type-safe resolvers with nested field resolution and computed fields
DataLoaders for batching — solving the N+1 problem in GraphQL
When to choose GraphQL vs REST for your API

19.1 Schema Design

A well-designed GraphQL schema is your API contract. Define Query for reads, Mutation for writes, and Subscription for real-time updates. Use the payload pattern for mutations: return a OrderPayload with both the result and potential errors, rather than throwing exceptions. This gives clients structured error handling. Design types to reflect your domain model — User has orders as a nested field, not a separate endpoint. This is the fundamental shift from REST: the client navigates relationships in a single query.

graphql

// GraphQL schema definition (schema.graphqls)
type Query {
    user(id: ID!): User
    users(filter: UserFilter, page: PageInput): UserConnection!
    order(id: ID!): Order
}

type Mutation {
    createOrder(input: CreateOrderInput!): OrderPayload!
    updateUserProfile(input: UpdateProfileInput!): User!
}

type Subscription {
    orderStatusChanged(orderId: ID!): OrderStatusUpdate!
}

type User {
    id: ID!
    name: String!
    email: String!
    orders(first: Int = 10): [Order!]!
    totalSpent: Money!
}

type Order {
    id: ID!
    user: User!
    items: [OrderItem!]!
    status: OrderStatus!
    total: Money!
    createdAt: DateTime!
}

input CreateOrderInput {
    userId: ID!
    items: [OrderItemInput!]!
    paymentMethod: PaymentMethod!
}

type OrderPayload {
    order: Order
    errors: [UserError!]
}

19.2 DGS Resolvers — Type-Safe Data Fetching

The Netflix DGS Framework maps your schema to Kotlin functions with annotations. @DgsQuery resolves top-level queries, while @DgsData resolves nested fields. The key insight: nested resolvers are only called when the client actually requests that field. If a client queries { user(id: 1) { name } }, the orders resolver is never called — this is GraphQL's efficiency advantage. Computed fields like totalSpent are resolved on-demand, keeping your User entity lean while providing rich derived data when clients need it.

kotlin

// Netflix DGS Framework — type-safe resolvers in Kotlin
@DgsComponent
class UserDataFetcher(
    private val userService: UserService,
    private val orderService: OrderService
) {
    @DgsQuery
    suspend fun user(@InputArgument id: Long): User? =
        userService.findById(UserId(id))

    @DgsQuery
    suspend fun users(
        @InputArgument filter: UserFilter?,
        @InputArgument page: PageInput?
    ): UserConnection =
        userService.findAll(filter, page?.toPageable())
            .toConnection()

    // Nested resolver — called only when "orders" field is requested
    @DgsData(parentType = "User", field = "orders")
    suspend fun ordersForUser(
        dfe: DgsDataFetchingEnvironment,
        @InputArgument first: Int?
    ): List<Order> {
        val user = dfe.getSource<User>()
        return orderService.findByUserId(user.id, limit = first ?: 10)
    }

    // Computed field — resolved on demand
    @DgsData(parentType = "User", field = "totalSpent")
    suspend fun totalSpent(dfe: DgsDataFetchingEnvironment): Money {
        val user = dfe.getSource<User>()
        return orderService.calculateTotalSpent(user.id)
    }
}

19.3 DataLoaders — Solving N+1 in GraphQL

The N+1 problem in GraphQL is especially insidious because the server doesn't control which fields are requested. A query like { orders { user { name } } } would trigger one query for orders, then one query per order to fetch the user — classic N+1. DataLoader solves this by collecting all user IDs requested in a single GraphQL execution and batching them into one query. Instead of N individual findById calls, you make a single findByIds(batch) call. This is mandatory for any production GraphQL API.

kotlin

// DataLoader — solves N+1 in GraphQL
@DgsComponent
class UserDataLoaderRegistrar {

    @DgsDataLoader(name = "users")
    val userLoader = MappedBatchLoaderWithContext<Long, User> { userIds, env ->
        // Called ONCE with all user IDs collected from the query
        // Instead of N individual queries, we make 1 batch query
        val users = userService.findByIds(userIds.toList())
        users.associateBy { it.id.value }
    }
}

// Using DataLoader in a resolver
@DgsComponent
class OrderDataFetcher {

    @DgsData(parentType = "Order", field = "user")
    suspend fun userForOrder(dfe: DgsDataFetchingEnvironment): CompletableFuture<User> {
        val order = dfe.getSource<Order>()
        val loader = dfe.getDataLoader<Long, User>("users")
        return loader.load(order.userId.value)
        // All user loads are batched into a single query
    }
}

// Query: { orders { id, user { name } } }
// Without DataLoader: 1 query for orders + N queries for users
// With DataLoader:    1 query for orders + 1 batch query for users

Interview Deep Dive

Q: What are the main trade-offs between GraphQL and REST?
A: GraphQL advantages: no over/under-fetching, strong typing, single endpoint, great for mobile (bandwidth-sensitive). GraphQL disadvantages: harder to cache (HTTP caching works on URLs, not POST bodies), complex authorization (field-level access control), potential for expensive queries (depth/complexity limits needed). Rule of thumb: use GraphQL for client-facing APIs with diverse consumers (web, mobile, partners), REST for simple CRUD APIs and machine-to-machine communication where the query shape is fixed.

Key Takeaway

GraphQL is not a replacement for REST — it's an alternative with different trade-offs. Always implement DataLoaders for batch resolution, enforce query depth and complexity limits to prevent abuse, and use the DGS framework for type-safe Kotlin integration. The combination of GraphQL for external APIs and gRPC for internal services gives you the best of both worlds.

Chapter 20: Behavioral & System Design Framework

Expert

At the Staff/Principal level, system design interviews are the most heavily weighted portion of the loop. You're not just evaluated on technical knowledge — interviewers assess your ability to structure ambiguity, make reasoned trade-offs, and communicate complex architectures clearly. This chapter provides a repeatable 5-step framework that works for any system design question, from URL shorteners to real-time chat systems. The framework forces you to start with requirements (not solutions), estimate scale, design high-level components, deep-dive into critical paths, and explicitly address bottlenecks and failure modes.

What you will learn

The 5-step system design framework: Requirements → Estimation → High-Level → Deep Dive → Trade-offs
Back-of-the-envelope capacity estimation with real numbers
How to identify and address bottlenecks, single points of failure, and hot spots
Behavioral framing: how to communicate technical decisions to non-technical stakeholders

20.1 The 5-Step Framework

Every system design interview should follow this structure: (1) Requirements clarification — ask what the system does, what scale it operates at, and what constraints exist. Never jump straight to "I'd use Kafka and Redis." (2) Capacity estimation — calculate QPS, storage needs, and bandwidth. This shows you think about scale concretely, not abstractly. (3) High-level design — draw the architecture: client → load balancer → API servers → cache → database. Identify the 3-5 core components. (4) Deep dive — pick the most interesting/challenging component and design it in detail: database schema, sharding strategy, caching policy, consistency model. (5) Trade-offs and bottlenecks — proactively identify single points of failure, hot keys, and what breaks at 10x scale.

kotlin

// System Design Interview — structured approach
// Use this framework for any system design question

// Step 1: Requirements clarification (2-3 minutes)
// - Functional: What does the system DO?
// - Non-functional: Scale, latency, availability targets
// - Constraints: Budget, team size, timeline

// Step 2: Capacity estimation (2 minutes)
// Example: Design a URL shortener
val dailyUsers = 100_000_000L // 100M DAU
val writesPerDay = dailyUsers * 0.1 // 10M new URLs/day
val readsPerDay = dailyUsers * 10   // 1B reads/day (100:1 read:write)
val writeQps = writesPerDay / 86400 // ~115 writes/sec
val readQps = readsPerDay / 86400   // ~11,500 reads/sec
val storagePerYear = writesPerDay * 365 * 500 // ~1.8TB/year (500 bytes/URL)

// Step 3: High-level design (5 minutes)
// Draw the architecture: Client → LB → API → Cache → DB
// Identify core components and data flow

// Step 4: Deep dive into components (15 minutes)
// - Database choice and schema
// - Caching strategy (write-through, write-behind)
// - Sharding / partitioning strategy
// - Consistency model (strong vs eventual)

// Step 5: Address bottlenecks and trade-offs (5 minutes)
// - Single points of failure
// - Hot spots / hot keys
// - What happens at 10x scale?

Real-World Scenario

Design a URL shortener: With 100M DAU, you'd calculate ~115 writes/sec and ~11,500 reads/sec (100:1 read:write ratio). The high-level design is: Client → LB → API (generates short code) → Redis cache (hot URLs) → PostgreSQL (persistent storage). For the deep dive, discuss: Base62 encoding for short codes, cache-aside pattern for reads, database sharding by hash of the short code, and CDN for geographic distribution. Trade-offs: strong consistency (reads-after-write) vs eventual consistency (faster but stale).

20.2 Capacity Estimation Cheat Sheet

Memorize these numbers for quick estimation in interviews. Latency: L1 cache: 1ns, RAM: 100ns, SSD random read: 150μs, HDD seek: 10ms, round-trip same datacenter: 0.5ms, round-trip cross-country: 70ms. Throughput: a single PostgreSQL instance can handle ~10,000-50,000 simple queries/sec, Redis handles ~100,000 ops/sec, a single Kafka broker handles ~100,000 messages/sec. Storage: 1 tweet = ~250 bytes, 1 URL = ~500 bytes, 1 user profile = ~1KB, 1 image thumbnail = ~50KB, 1 high-res photo = ~2MB.

Component	Throughput	Latency
PostgreSQL (indexed)	10-50K qps	1-5ms
Redis	100K ops/sec	sub-ms
Kafka (per broker)	100K msg/sec	2-5ms
Nginx (HTTP)	50K req/sec	sub-ms
gRPC (unary)	30K req/sec	1-2ms

Interview Deep Dive

Q: How do you handle a behavioral question about a project that failed?
A: Use the STAR framework (Situation, Task, Action, Result) but add a Lessons Learned section. Staff-level interviewers want to see self-awareness and growth. Example: "We chose microservices too early (Situation/Task). I advocated for service decomposition without considering our team's operational maturity (Action). Deployments became fragile, and we spent 60% of sprint time on infrastructure (Result). I led the migration back to a modular monolith, which cut deployment failures by 80% (Lesson: organization and maturity dictate architecture, not technical elegance)."

*Signal vs noise in system design: Interviewers care about your decision-making process, not your ability to name-drop technologies. Saying "I'd use Redis for caching because the read:write ratio is 100:1 and we need sub-millisecond reads for the hot path" is 10x stronger than "I'd use Redis because it's fast." Always explain why you chose something, not just what you chose.

Key Takeaway

System design is about structured thinking under ambiguity. Follow the 5-step framework, lead with requirements not solutions, and always discuss trade-offs explicitly. The best candidates don't present a "perfect" design — they present a reasoned design with clearly articulated trade-offs and alternatives they considered.

Appendix A: Recommended Stack (2026)

A production-ready stack for building high-performance, maintainable backend services in Kotlin. Each choice reflects the state of the art in 2026.

Language

Kotlin 2.2

Framework

Spring Boot 4.x

Build

Gradle (Kotlin DSL)

Persistence

Spring Data JPA + Exposed

Concurrency

Kotlin Coroutines

Testing

JUnit 5, MockK, Testcontainers

Observability

OpenTelemetry, Micrometer

Messaging

Apache Kafka

Cache

Redis / Valkey

API

REST + gRPC (internal)

CI/CD

GitHub Actions + ArgoCD

Container

GraalVM Native + Docker