MongoDB for Event Logs: The Snowflake → Mongo Story

Every valuation this service computes produces an audit record: which providers were called, what each one returned, which rules fired, what the final decision was, and — increasingly — metadata nobody anticipated when the record shape was first designed, like a fraud-risk score added last quarter or a `promotion_code` field added the quarter before that. For a long time, this data followed a familiar path in this organization: writes accumulated somewhere transactional, a nightly batch job extracted them, transformed them into the warehouse's expected shape, and loaded them into Snowflake, where analysts and dashboards finally got to see them.

That pipeline is a completely reasonable design for the problem it was built for — heavy, ad hoc analytical queries across years of history, the kind Snowflake is genuinely excellent at. But it has two costs that mattered more and more as this service grew, and they are worth stating precisely rather than hand-waving as 'batch is slow.' First: visibility lag. If a valuation happens at 9am and the batch job runs at 2am the next day, a support engineer investigating a dealer's complaint at 11am that same morning cannot see the record in the warehouse at all yet — they are debugging blind, or falling back to grepping application logs, for up to a day.

Second, and more corrosive over time: schema rigidity as a cross-team contract. Snowflake's ingestion expected a defined, agreed-upon shape. Adding a new field to the audit event — the fraud score, the promotion code, eventually a field for which circuit breaker state the request observed — meant coordinating a schema change with the data warehouse team, updating the ETL transform, and often waiting for a release window that had nothing to do with this service's own release cadence. The event payload, which is naturally evolving as the business asks new questions, was chained to the pace of a team and a pipeline that had every reason to be conservative about changing a shared contract.

Moving the audit trail to a `valuation_event` MongoDB collection, written directly by `VehicleValuationService` at the moment a valuation completes, addresses both costs at their root. Visibility lag disappears because the write happens in the same request path that computed the valuation — the record exists in Mongo within milliseconds, not by the next day's batch window. Schema rigidity disappears because MongoDB does not enforce a fixed shape across documents in a collection: this quarter's documents can carry `fraudRiskScore`, last quarter's did not, and neither the collection nor any other consumer needs to be told about the addition in advance. A new field is just a new field in the next document written; nothing has to migrate.

This is not a free upgrade, and it is worth being honest about what got traded away. Snowflake's columnar storage and mature SQL analytics are built for exactly the queries a document store handles poorly — aggregate a metric across 400 million events, join it against three other tables, and do it in seconds. MongoDB's document model is built for the opposite: fetch this event by ID, or by a handful of predictable filters (VIN, date range, provider), fast, at write-time scale. Ask it to do heavy cross-collection analytical aggregation at the volume this audit log reaches, and it will do it — but noticeably worse than a warehouse purpose-built for that job.

The resolution is not 'pick one forever' — it is decoupling the write path from the analytics path. `VehicleValuationService` writes to MongoDB synchronously (or, as Module 4 will show, asynchronously via a fire-and-forget coroutine, which removes even the small latency cost of that write from the response) because that is the operational, low-latency need. A downstream ETL job can still sync MongoDB into a warehouse for the analysts who need the heavy aggregate queries — but that sync happens on its own schedule, decoupled entirely from whether a dealer is waiting on a valuation response, and a change to the warehouse's ingestion process no longer blocks or slows this service's own evolution.

The shape of that decision generalizes past this one service: when a write path's job is 'make this fact durable and queryable right now, in a shape that will keep changing,' and an entirely separate concern is 'let analysts run heavy historical queries across everything that has ever happened,' those two jobs deserve two different stores connected by an explicit, asynchronous pipeline — not one store straining to do both.

// A document in the valuation_event collection.
// Note fraudRiskScore and promotionCode: fields added in later quarters
// that simply did not exist on older documents — no migration required.
{
  "_id": ObjectId("64f1c2a1e4b0f5a1d2c3b4a5"),
  "vin": "1HGCM82633A004352",
  "requestedAt": ISODate("2026-08-22T14:03:11Z"),
  "providerResponses": [
    { "provider": "BLACK_BOOK", "value": 18250, "latencyMs": 340 },
    { "provider": "CARFAX", "value": 17980, "latencyMs": 610 },
    { "provider": "SP_VIS", "value": null, "error": "TIMEOUT" }
  ],
  "rulesApplied": ["GUARDRAIL_BRANDED_TITLE", "BOOST_LOW_MILEAGE"],
  "finalOffer": 17500,
  "fraudRiskScore": 0.12,
  "promotionCode": "SUMMER26"
}

@Document(collection = "valuation_event")
data class ValuationEvent(
    @Id val id: String? = null,
    val vin: String,
    val requestedAt: Instant,
    val providerResponses: List<ProviderResponse>,
    val rulesApplied: List<String>,
    val finalOffer: BigDecimal,
    val fraudRiskScore: Double? = null,      // added later; older docs simply omit it
    val promotionCode: String? = null,       // added later too
)

interface ValuationEventRepository : MongoRepository<ValuationEvent, String> {
    fun findByVinOrderByRequestedAtDesc(vin: String): List<ValuationEvent>
}

// Written directly in the request path — no nightly batch job in between.
eventLog.save(ValuationEvent.from(vin, rulesApplied = firedRules, finalOffer = result.offer))