Hermes Wiki
Developer/Compliance/ComplianceFrameworks/Challenges/design-a-kyc-aml-onboarding-pipeline-for-a-fintech-platform

Design a KYC/AML Onboarding Pipeline for a Fintech Platform

Scenario prompt

You're the architect for a neobank/exchange onboarding new users across the US, EU, and UK. Before an account can hold funds, it must pass Know Your Customer (identity verification) and Anti-Money-Laundering (sanctions/PEP screening) checks. Requirements:

  • Verification runs through multiple third-party vendors — document verification, biometric liveness, sanctions-list screening — each with its own latency profile and failure modes
  • Regulators require a fully auditable decision trail: what data was checked, against which list version, by which vendor, at what time, and who or what approved or rejected the applicant
  • False positives (legitimate users blocked) need a human review/appeal path; false negatives (bad actors let through) carry real regulatory fines
  • Sanctions lists (OFAC, EU, UN) update continuously — a user cleared today can appear on a new list tomorrow, which means re-screening the entire existing user base, not just new signups
  • Jurisdictions disagree on required checks and retention rules — GDPR's right-to-erasure sits in direct tension with AML's multi-year retention mandate

Mihir's attempt

[!todo] Write your own attempt here before reading the model solution below — how you'd orchestrate multiple unreliable vendors into one auditable decision, and how you'd resolve the GDPR-vs-AML-retention conflict.

Model solution

Model each applicant as a state machine, not a single synchronous call. submitted → documents_verified → screening_in_flight → decision → {approved, rejected, manual_review}. A single onboarding attempt fans out to several vendors that don't share a latency budget or a failure mode, so the orchestration has to tolerate partial failure and retries without corrupting the overall decision — the same shape of problem Design a Payment Reconciliation System solves for multi-party financial state. Each vendor call gets its own circuit breaker and backoff-with-jitter retry policy so one slow biometric vendor doesn't stall document verification, and the state machine only advances once all required checks report back (or time out into manual review).

Make the audit trail an immutable event log, not a mutable "status" column. A regulator asking "what did you check, against what list version, and why did you approve this account" needs a replayable history, not a snapshot. Append every vendor request/response, list-version hash, and decision rationale to an event-sourced ledger per applicant, the same pattern as Event Sourcing and CQRS — the current "approved" status is a derived projection of that log, never the source of truth itself. This is the same instinct behind Design an Audit Log System for a Regulated Trading Platform: provability requires the log to exist independently of whatever service currently reads it.

Treat sanctions-list updates as a diff-driven re-screening stream, not a nightly full re-run. Subscribe to vendor list-update feeds; on each update, compute the diff against the existing customer base and push only the affected users back through the same onboarding state machine as a re-screening event, fanned out via the org's event streaming backbone. Re-running the full customer base against every list update doesn't scale past a few hundred thousand users and doesn't need to — the list changed by a handful of entries, not by everyone.

Resolve GDPR-vs-AML by separating the PII from the decision metadata at the schema level, then applying different retention clocks to each. This mirrors Design a Cross-Service Data Deletion (Right to Be Forgotten) Pipeline's core move: isolate raw PII (documents, biometric templates) behind per-user encryption keys so it can be crypto-shredded once the shorter of "user requested erasure" and "AML retention window elapsed" is satisfied, while the non-PII decision record — which vendor, which list version, what the outcome was, with the PII reference nulled out — persists for the full regulatory retention period untouched. The regulator gets a permanent audit trail; the user's actual documents don't outlive their legal necessity.

Give ambiguous decisions a real human queue with an SLA, inside the same state machine. Vendor confidence scores that land in a gray zone shouldn't auto-reject — they should route to manual_review with the full evidence bundle attached, a review SLA tracked like any other operational metric, and a defined appeal path back into the state machine if the applicant disputes the outcome.

Gaps to revisit

  • What happens when two vendors disagree — one clears the applicant, another flags a partial name match? The model above routes ambiguity to manual review, but doesn't define a scoring/consensus policy across vendors.
  • Data residency: EU regulators increasingly require in-region processing, which can conflict with a US-based vendor's default data handling — does the architecture need per-region vendor routing, not just per-region retention rules?
  • Cost: re-screening even a small diff against a large existing customer base still means a nonzero per-check vendor fee at every list update — at what update frequency does this become a material line item worth negotiating into the vendor contract?

Engineering Lens

This is a distributed-orchestration problem wearing a compliance costume, and it's exactly the kind of system Mihir's Fintech/Capital Markets target roles live in. The technical shape — saga-style state machine, event-sourced audit trail, diff-driven re-processing — is generic distributed-systems judgment; what makes it Principal-level is recognizing which of those generic patterns a regulator will actually ask about in an audit, and designing so the answer is "the log proves it" rather than "we're pretty sure." Defending the crypto-shredding-plus-separate-retention-clock design in front of a compliance officer, and being able to say precisely what's deleted versus what's retained and why, is the difference between a system that survives its first regulatory exam and one that doesn't.

Hermes Wiki