GDPR Core Principles and Erasure Engineering
Concept
GDPR's engineering-relevant substance boils down to a handful of principles (Article 5): lawfulness/fairness/transparency, purpose limitation, data minimization, accuracy, storage limitation, integrity/confidentiality, and accountability. Most of these are policy-and-process concerns, but two produce concrete, hard engineering obligations: storage limitation (don't keep personal data longer than its stated purpose requires — see Data Retention and Deletion Policy for the mechanics) and Article 17, the right to erasure ("right to be forgotten") — a data subject can request their personal data be deleted, and the controller has to actually be able to do it, not just intend to.
Article 17 is not absolute; it carries real exceptions worth knowing precisely because they're where "just delete everything" goes wrong: processing necessary to comply with a separate legal obligation (Article 17(3)(b) — e.g., a tax-retention mandate that requires keeping a financial record for years), archiving in the public interest or for scientific/historical/statistical purposes, and the establishment, exercise, or defense of legal claims. This is exactly why a WORM-protected audit log (see WORM Storage and Immutability Guarantees) and GDPR erasure aren't actually in conflict as often as they look: the resolution is usually to erase or pseudonymize the personal-data fields within an audit event while leaving the immutable event shell (timestamp, action type, outcome) intact — the record survives, the identifying content doesn't.
The harder engineering problem is proving erasure actually happened. Soft delete (is_deleted = true) is not erasure under Article 17 — the data is still physically present and readable by anyone with database access, and regulators treat this as functional deletion at best, not verifiable erasure. Hard deletion solves that for the primary store, but breaks down at scale once personal data has propagated into backups, data warehouse exports, replicas, and (increasingly relevant) embedding stores — a 2026 arXiv study ("Ghost Vectors") found that soft-deleted embeddings remain reconstructible from HNSW vector-database indexes even after the source record is marked deleted, because the vector itself, not just the row, has to be actually removed. Selectively re-processing every backup snapshot to strip one subject's rows is often operationally infeasible at real data volumes.
Crypto-shredding is the practical answer for exactly that scale problem: encrypt each data subject's personal fields with a unique, per-subject encryption key, and treat "erasure" as destroying that key rather than hunting down and rewriting every copy of the data. Once the key is gone, every copy — including ones sitting untouched in old backups — becomes permanently unreadable ciphertext, without needing to locate and modify each copy individually. Spotify built exactly this as a production system, Padlock: a key-management service that issues and stores a keychain per user, which every downstream service queries to encrypt/decrypt that user's data; erasure is deleting the user's root key from Padlock, after which all of that user's data across every service and every backup becomes permanently unrecoverable. The EDPB hasn't formally endorsed crypto-shredding as satisfying Article 17 outright, but several EU data protection authorities have accepted it in practice where selectively deleting from petabyte-scale backups would otherwise be "manifestly disproportionate effort" under Article 17(1).
Tradeoffs
| Approach | Verifiable erasure | Handles backups/replicas | Engineering cost |
|---|---|---|---|
| Soft delete (flag only) | No — data is intact and readable, fails Article 17's actual bar | No — irrelevant, nothing is removed anywhere | Lowest |
| Hard delete from primary store only | Partial — satisfies the primary store, not the whole system | No — backups, warehouse exports, and replicas retain the data untouched | Low |
| Hard delete with cascading propagation to every known copy | Yes, if the propagation map is actually complete | Yes, but only as far as the map goes — a missed downstream copy is a silent failure | High — requires a maintained lineage map of every place the data flows |
| Crypto-shredding (per-subject key, destroy on erasure) | Yes — ciphertext without the key is unrecoverable, provable by showing the key is gone | Yes, including old backups nobody re-processes | High upfront (per-subject key management infrastructure) but low marginal cost per erasure request thereafter |
When to use / when not to
- Use crypto-shredding once backups, data warehouse copies, or replicated stores make hunting down and rewriting every physical copy of a subject's data operationally infeasible — this is the regime that scales to real erasure-request volume without a bespoke propagation job for every request.
- Hard deletion from a single mutable primary store is sufficient when the system genuinely has no long-lived backups or downstream copies to worry about — don't build per-subject key management infrastructure to solve a propagation problem you don't have.
- When personal data lives inside an otherwise-immutable audit or WORM record, erase/pseudonymize the personal-data fields specifically rather than trying to delete the whole record — the retention and erasure obligations apply to different parts of the same event.
- Don't rely on soft delete as an erasure mechanism for anything a data subject can invoke Article 17 against — it doesn't meet the bar and won't survive a regulator's actual test of "show me the data is gone."
Common pitfall
Declaring a deletion request complete because the primary application database no longer returns the row, without accounting for the same personal data sitting untouched in nightly backups, a data-warehouse export, or — increasingly — a vector embedding in a semantic-search index. The Ghost Vectors finding on HNSW indexes is a sharp example: a system can pass its own "is the row gone" test while the embedding derived from that row remains fully reconstructible from the index structure itself. A deletion workflow is only as complete as its map of every place the data actually went — see Data Lineage Tracking for how that map gets built and kept current.
Engineering Lens
The design-review question worth asking isn't "do we support account deletion" — most systems technically do — it's "if a regulator asked us to prove a specific subject's data is now genuinely unrecoverable, including from every backup taken before the request, could we produce that proof, and how long would it take." Crypto-shredding's real value in that conversation is that the proof becomes trivial (show the key is gone) instead of an audit of every storage system the company operates. That's a materially different engineering commitment than a soft-delete flag, and it's worth pricing in during initial data-model design — retrofitting per-subject encryption onto data that's already been flowing unencrypted into backups for years is a much larger project than building it in from the start.
Sources
- Art. 17 GDPR – Right to erasure ('right to be forgotten')
- Scalable User Privacy — Spotify Engineering
- Ghost Vectors: Soft-Deleted Embeddings Remain Reconstructible in HNSW Vector Databases — arXiv