Design a Cross-Service Data Deletion (Right to Be Forgotten) Pipeline
Scenario prompt
Your platform's user data has fanned out across dozens of microservices, each with its own datastore — a primary DB, caches, search indices, an analytics warehouse, nightly backups, and a few third-party processors. Legal requires that on a verified deletion request (GDPR Article 17, CCPA), all personal data tied to that user is deleted or anonymized within a defined SLA (commonly 30 days), including in backups and downstream pipelines. Requirements:
- No individual team can be trusted to remember, unprompted, to wire in deletion handling for every new service they ship
- Deletion completion must be provable to an auditor or regulator, not just asserted
- Backups can't be surgically edited on demand, so there needs to be a defined mechanism that still satisfies the SLA
- Deleting a user's PII must not silently corrupt aggregate, non-PII analytics that are legitimately allowed to persist
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — how you'd guarantee every future service participates in deletion without relying on institutional memory, and how you'd handle backups you can't edit in place.
Model solution
Make deletion a registered contract every service must join, not a courtesy every team remembers. Stand up a central PII catalog/registry that every service must declare into as part of onboarding a new datastore — what tables/fields hold PII for a given user, keyed by a canonical user ID. This is the same governance-by-registration idea as Netflix's Nebula ArchRules: instead of trusting every team to remember a cross-cutting rule, you make the rule mechanically enforced — a service that stores PII but never registered in the catalog fails a CI/deploy-time check, the same way an unregistered architecture rule violation would.
Drive deletion through an event-driven fanout with tracked acknowledgment, owned by a central state machine. A deletion-request service owns the workflow: pending → fanned-out → per-service ack → complete. On a verified request, it publishes a "user X deletion requested" event; every service registered in the PII catalog subscribes and is contractually required to delete/anonymize its data and emit an ack within the SLA window. This mirrors the durable-fanout structure of Event Sourcing and CQRS — the event log is the source of truth for "was this actually requested," independent of whether any one consumer was up at the time.
Solve backups with crypto-shredding instead of trying to edit frozen snapshots. You cannot practically find-and-delete one user's rows inside a nightly backup snapshot, and you shouldn't try. Instead, encrypt PII per-user (or per-tenant, at minimum per some reasonably small key-scope) at write time with a key managed independently of the data itself, per Encryption: At Rest, In Transit, and In Use. Deletion becomes "destroy that user's key" rather than "edit every historical backup" — the ciphertext left behind in old snapshots is permanently unrecoverable the moment the key is gone, satisfying the deletion requirement without ever touching immutable backup storage. Backup retention policy (per Data Storage Tiering and Lifecycle Policies) still bounds how long any backup exists at all, independent of this mechanism.
Separate PII from aggregate analytics at the schema level, before deletion ever has to reason about it. If a user's PII and the anonymized/aggregated metrics derived from their activity live in the same row or table, a deletion job either has to selectively edit fields (fragile, easy to get wrong) or over-delete (destroying legitimate business history). Designing ingestion so PII fields are isolated from derived aggregate fields from day one — a user_pii table joined by ID rather than PII columns scattered across every analytics table — means deletion is a clean drop of one well-scoped thing, not a surgical edit across the schema.
Gaps to revisit
- Third-party sub-processors who received the data legitimately (a payment processor, an email vendor) are a contractual problem, not a technical one — how does the deletion-request service track and trigger their deletion obligations, and what's the audit story when you're depending on another company's word?
- What happens when a registered service is unreachable, deploys a breaking change to its consumer, or simply fails to ack within the SLA window — does the central state machine escalate to a human, retry indefinitely, or breach the SLA and require legal notification?
- The PII catalog itself needs versioning discipline as schemas evolve — a field added six months after a service registers is invisible to the deletion contract unless registration is enforced continuously (e.g., in CI), not just once at onboarding.
Principal Engineer Lens
This is fundamentally an organizational-complexity problem wearing a technical costume: the hard part isn't deleting a row, it's guaranteeing that forty independently-shipping teams all honor a cross-cutting obligation without a central authority manually chasing each one down. That's the same class of problem as enforcing security policy or architectural conventions across a large org — and the fix is the same pattern each time: make the rule mechanically enforced at a chokepoint (CI, deploy gate, service registration) rather than a wiki page everyone's supposed to read. Defending "we built a self-service compliance framework instead of a one-off deletion script per request" in front of a compliance or legal stakeholder is a genuinely Principal-level conversation — it's the difference between a system that scales with headcount and one that requires linear manual effort per new service forever.
Reel Script
Setup: A user asks you to delete their data under GDPR. Your platform has forty microservices, each with its own database, and nobody has a full list of which ones actually touch that user's PII. How do you make deletion provably complete instead of best-effort?
Concept walkthrough: Explain the PII catalog as an enforced registration contract, the deletion-request state machine driving an event-driven fanout with tracked per-service acknowledgment, and crypto-shredding as the answer to "how do you delete from a backup you can't edit."
Real example tie-in: Walk through the Nebula ArchRules governance model — how Netflix enforces architectural rules across thousands of repos mechanically rather than by asking teams nicely — and map that same "enforce at a chokepoint, not by memory" logic onto PII registration.
Tradeoffs & alternatives: Contrast crypto-shredding (fast, clean, requires per-user/tenant key granularity designed in from the start) against physically re-writing backups (slow, error-prone, sometimes impossible against immutable storage) — and why almost every mature platform lands on crypto-shredding once they've tried the alternative once.
Principal Engineer takeaway: Compliance systems that depend on institutional memory don't scale past a handful of services — they scale past forty only if the obligation is enforced by tooling at the same chokepoint every service already has to pass through to ship at all.