Design an Audit Log System for a Regulated Trading Platform
Scenario prompt
Design the audit logging system for a capital markets platform (think an order-management or trading system) subject to regulatory retention and tamper-evidence requirements (SEC Rule 17a-4-style). Requirements:
- Every state-changing action — order placed, order modified, order cancelled, permission changed, trade booked — must produce an immutable audit record: who, what, when, and the before/after state
- Records must be provably tamper-evident — an auditor or regulator must be able to detect if any historical record was altered or deleted after the fact
- Retention is measured in years, but the hot query path (a compliance officer investigating a specific trade from this morning) needs sub-second lookups
- The audit pipeline must never be the reason a trade fails — logging cannot sit in the critical path of order execution
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — how you'd keep audit logging off the critical execution path while still guaranteeing no state-changing action goes unlogged, and how you'd make old records provably tamper-evident.
Model solution
Emit audit events asynchronously from the write path, but make emission itself durable and mandatory, not best-effort. The trading engine can't block an order on a logging call — that violates the "audit pipeline must never fail a trade" constraint and turns a compliance system into an availability liability. The fix is the same pattern as Event Sourcing and CQRS: the state-changing action and its audit record are written together, atomically, at the source (e.g., in the same database transaction, or via a transactional outbox published to a durable log like Kafka), and a separate consumer asynchronously ships that record into the long-term audit store. This guarantees no action is missed without making the audit store's latency or availability part of the trading engine's critical path.
Make records tamper-evident with hash chaining, not just access control. Access control (who can write, who can read) prevents unauthorized changes but doesn't let an auditor detect an authorized insider quietly editing history. The standard fix is a hash chain: each audit record includes a hash of the previous record plus its own content, so altering any historical record breaks every subsequent hash and is detectable by recomputing the chain. Periodically anchoring the current chain hash to an external, independent system (a separate write-once store, or even a public timestamping service) means even someone with full database access can't rewrite history without the discrepancy showing up against that external anchor — this is the same defense-in-depth instinct as Defense in Depth, layered specifically against a privileged-insider threat model rather than an external attacker.
Store on genuinely immutable, WORM-class media for the regulatory retention window. SEC 17a-4-style rules specifically require write-once-read-many storage, not just "we promise not to delete it" — object storage with object-lock/retention-lock (e.g., S3 Object Lock in compliance mode) satisfies this without hand-rolling custom hardware. Data storage tiering (per Data Storage Tiering and Lifecycle Policies) then splits the retention window: a hot tier (fast disk/DB, indexed by trade ID, account, timestamp) covers recent activity for sub-second compliance-officer lookups, while a cold WORM tier holds the multi-year archive at lower cost, with the hash chain spanning both so a record moving tiers doesn't lose its tamper-evidence.
Index deliberately for the actual investigation pattern, not just chronological insertion order. A compliance investigation almost never starts "show me everything from 9:31am" — it starts from a specific trade ID, account, or user, then expands outward in time. Designing the hot-tier index around those lookup keys (rather than defaulting to append-only chronological scan) is what actually delivers the sub-second requirement; a hash-chained log that's fast to write but slow to query on the dimension investigators actually use has solved only half the problem.
Gaps to revisit
- Cross-region regulatory regimes — a global platform may face conflicting retention/deletion rules (SEC 17a-4 retention vs. GDPR right-to-erasure) for the same underlying activity, and reconciling those isn't a purely technical problem
- Key management for the hash chain and any signing — if the signing key itself is compromised, the tamper-evidence guarantee is only as strong as key custody, which pushes part of this problem into HSM/KMS territory
- Audit-of-the-audit-system — who watches the watcher, and how do you prove the audit pipeline itself wasn't paused or degraded during an incident window
Engineering Lens
This challenge is a clean illustration of a pattern that recurs anywhere compliance meets distributed systems: the naive approach (log synchronously, trust access control) satisfies neither the "never block the critical path" nor the "provably tamper-evident" requirement, and the real design is two separable concerns — guaranteed emission (solved with durability/transactional patterns) and guaranteed immutability (solved with hash chaining plus WORM storage), not one mechanism doing both jobs. Being able to decompose a fuzzy compliance requirement into the specific technical guarantees that satisfy it, and to explain why access control alone is insufficient against an insider threat model, is exactly the kind of reasoning that reads as Principal-level judgment in a capital-markets or fintech architecture review — directly relevant given Mihir's targeting of Fintech and Capital Markets roles.