Design a Real-Time Fraud Detection System
Scenario prompt
Design a fraud detection system that sits in the payment authorization path for a card-not-present e-commerce processor. It needs to:
- Score every transaction for fraud risk in well under 100ms, since it sits synchronously in the checkout/authorization flow
- Combine fast rule-based checks (velocity limits, blocklists, geo mismatches) with a machine-learned risk model, without either one becoming the bottleneck
- Keep learning from new fraud patterns without a redeploy — fraud rings adapt within hours, not release cycles
- Decide what happens to a transaction when the scoring pipeline itself is degraded or unavailable, given that both "block everything" and "approve everything" are actively bad outcomes
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — how you'd split rules vs. ML in the hot path, how you'd keep the model current, and what you'd do when scoring itself is degraded.
Model solution
Tiered scoring: a fast rule/feature layer in front of the ML model, not instead of it. The hot path runs cheap, deterministic checks first — velocity limits (too many attempts from one card/device/IP in a short window), static blocklists, and hard geo/BIN mismatches — computed off features already sitting in a low-latency feature store (recent transaction counts, device fingerprint history) rather than queried live from a transactional database. Only transactions that clear the cheap checks go to the ML model for a nuanced score. This keeps the median-case latency low, since most legitimate transactions never touch the expensive path, while still catching the classic patterns rules are good at (rules are precise and explainable; ML is better at the subtle, evolving patterns rules can't be hand-written for).
The model serves pre-computed features, not live joins, to hit the latency budget. A synchronous call that queries a transactional database for "this card's spending pattern over 90 days" at request time is a latency and load disaster. Instead, a streaming pipeline (Kafka/Flink-style) continuously maintains a feature store keyed by card/device/user, updated asynchronously as transactions happen, so the synchronous scoring call is a fast key-value read plus a model inference — not a live aggregation.
Model freshness is solved by decoupling "deploy a new model" from "deploy new code." The scoring service loads a model artifact from a model registry and reloads it on a schedule (or on a push signal) — retraining and redeploying a model is a data/ML-ops pipeline concern, separate from the service's code deploy cycle. This is what lets the model adapt to a fraud ring's new pattern within hours: retrain on fresh labeled data, publish a new artifact version, and the serving layer picks it up without a service redeploy. Some teams add a fast-follow rule layer specifically to react to a live fraud spike before a retrained model is even ready — rules as the hours-timescale lever, model retraining as the days-timescale lever.
Degraded-mode behavior is a graduated response, not a binary fail-open/fail-closed. Unlike a rate limiter, "fail open" (approve everything) during an outage is a direct financial-loss risk, and "fail closed" (decline everything) is a direct revenue and customer-trust hit — neither is acceptable as a blanket default. The pragmatic answer is a graduated fallback: if the ML model is unavailable, fall back to the rule layer alone (still meaningful protection, just less precise); if the whole scoring pipeline is down, fall back to a conservative static policy — e.g., approve under a low transaction-amount threshold, route above-threshold or highest-risk-BIN transactions to step-up authentication or manual review, rather than either extreme. This decision should be a named, tested runbook, not an accident of whatever the client SDK does on timeout.
Gaps to revisit
- How do you evaluate a new model version before it's live — shadow-mode scoring (compute the new model's score alongside the live one, compare outcomes, without acting on it) is standard, but what's the bar for promoting it?
- Adversarial adaptation: once a fraud ring learns the rule thresholds (by probing), how often do rules need to change, and how do you avoid tipping your hand by being too consistent?
- False positives have a real cost too — a legitimate customer wrongly declined is lost revenue and trust. How do you measure and budget for that side of the tradeoff, not just the fraud-caught side?
Principal Engineer Lens
This is a genuinely different flavor of "degraded mode" design than most resilience problems: the usual fail-open/fail-closed binary (rate limiter, secrets cache) breaks down here because both extremes carry direct financial and trust cost, which forces a graduated, policy-driven fallback instead of a single default. Being able to articulate that distinction — "this isn't a fail-open-or-closed question, it's a risk-tiered degraded-mode question" — is exactly the kind of framing that reads as Principal-level judgment in a design review, because it shows the tradeoff was actually reasoned through rather than pattern-matched from a simpler problem. This maps directly onto Mihir's stated Fintech/Capital Markets target: the same tiered-scoring, feature-store, and graduated-fallback shape shows up in AML transaction monitoring and trading pre-trade risk checks, not just card fraud.
Reel Script
Setup: Every card swipe or online checkout gets scored for fraud in the time it takes to blink — how do you make that call fast, keep it current, and not do catastrophic damage the day the scoring system itself has a bad day?
Concept walkthrough: Start with why the hot path can't do a live database query per transaction — walk through the tiered design: cheap rule checks first, off a pre-computed feature store, with the ML model only invoked for what clears the cheap layer. Explain why the model reads pre-aggregated features instead of computing them live, and how that's what makes sub-100ms latency achievable at all.
Real example tie-in: Walk through a fraud ring adapting its pattern within hours — explain how rules can be hand-updated fast as a stopgap while a model retrain-and-redeploy pipeline (separate from the service's code deploys) catches up over a slower timescale.
Tradeoffs & alternatives: Contrast the usual fail-open/fail-closed binary from other resilience problems with why fraud scoring needs a graduated degraded-mode policy instead — approve under a threshold, step up or hold above it — since both blanket extremes carry real financial cost here.
Principal Engineer takeaway: The interesting design move isn't rules-vs-ML, it's recognizing when a problem doesn't fit the simple fail-open/fail-closed pattern and needs a genuinely graduated response instead — and being able to name that distinction explicitly is what separates a reasoned design from a copied one.