Developer/RecordsAudit/AuditLogging/Fundamentals/audit-logging-and-tamper-evident-trails

Audit Logging and Tamper-Evident Trails

Concept

An audit log is a distinct artifact from an application log, even though both are "records of things that happened." An application log exists to help an engineer debug — it can be sampled, rotated aggressively, or lost without lasting consequence. An audit log exists to answer, months or years later and under adversarial scrutiny (a regulator, an auditor, opposing counsel in a dispute), the question "who did what, to what, and when" — and its value depends entirely on the answer being trustworthy even if the person who performed the action would prefer it weren't recorded. That requirement drives every design choice: an audit log must capture actor, action, target, and timestamp for every sensitive operation (admin actions, permission changes, access to PII or payment data), and it must be append-only — written to a destination that disallows edits and deletes, not merely a database table an application happens not to update.

Append-only alone isn't sufficient, because "the application doesn't expose a delete button" and "deletion is actually impossible" are different guarantees — anyone with direct database or storage access (including the same administrators the log exists to hold accountable) can still edit rows or overwrite objects unless the storage layer itself enforces immutability. Two complementary techniques close that gap. Storage-level immutability — S3 Object Lock in WORM (write-once-read-many) mode, or an equivalent compliance-mode bucket — makes deletion or modification fail at the storage layer regardless of who's asking, for a configured retention period. Tamper-evidence via hash chaining — each log entry's hash incorporates the hash of the previous entry, so altering or removing any entry breaks every hash after it — doesn't prevent deletion, but makes it detectable: a verifier who recomputes the chain and finds a mismatch knows tampering occurred, even without knowing exactly what changed. AWS CloudTrail combines both ideas concretely: it delivers a signed digest file every hour containing the SHA-256 hash of each log file delivered that hour, so if a log file is altered or deleted after delivery, recomputing its hash against the digest reveals the mismatch — and the digest files themselves are separately protected, so an attacker would need to compromise two artifacts, not one, to hide the tampering.

Tradeoffs

Approach	Tamper resistance	Operational cost	Failure mode
Mutable log table (regular DB rows)	None — any writer with UPDATE/DELETE access can alter history silently	Lowest — no special infrastructure	A compromised admin account or a bad migration can rewrite the record with no trace
Append-only table/API, no storage enforcement	Relies entirely on discipline/access control; a privileged actor (DB admin, cloud root) can still bypass the application layer	Low-moderate	"Append-only" is a convention, not a guarantee — anyone with infra-level access defeats it
Immutable storage (WORM / Object Lock)	Strong against deletion/edit for the lock duration, enforced by the storage layer itself	Moderate — retention periods must be set correctly up front; locked data can't be corrected even to fix a genuine bug	Can't be undone even when you legitimately need to (a lock set too long, or on the wrong bucket)
Hash-chained entries + periodic external anchoring	Doesn't stop deletion, but makes any tampering mathematically detectable on verification	Moderate — needs a verification job and somewhere trustworthy to anchor periodic root hashes	Detects tampering after the fact; doesn't prevent an outage or gap if the whole chain is deleted before anyone verifies it

The strongest real-world designs combine the last two: immutable storage to make casual tampering impossible, plus hash chaining and periodic external anchoring (as CloudTrail's digest files do) so that even a sophisticated attempt to compromise the storage layer itself leaves a verifiable trace.

When to use / when not to

Use dedicated audit logging for any admin action, permission/role change, or access to sensitive data (PII, payment details, health records) — anything that could become the subject of a dispute, security investigation, or compliance audit.
Use it at the boundary of privileged operations specifically — a support agent viewing a customer's payment method, an engineer running a manual database update, a manager approving a refund — not for routine, low-stakes reads that carry no accountability question.
Don't conflate this with general application logging (see Structured Logging) — application logs can be sampled, rotated, and discarded for cost reasons; an audit log's retention and immutability requirements are usually driven by legal/compliance minimums, not operational convenience.
Don't over-apply immutable storage to logs that don't need it — WORM buckets are more expensive and less flexible (data locked in error can't be corrected), so reserve them for genuinely accountability-critical events, not every log stream by default.

Common pitfall

Giving the same IAM role or database credentials that can perform the sensitive action the ability to also delete or alter the record of having performed it. If a production administrator's credentials can both issue a refund and delete the CloudTrail-equivalent log of that refund, the audit trail provides no real accountability — it only looks like it does until the one time it matters. The fix is structural, not procedural: the storage location for audit logs must be provisioned with permissions separate from, and not writable-by, the roles whose actions it records — a genuine separation of duties, enforced by IAM policy rather than by asking people not to abuse their access.

Engineering Lens

The question that actually matters in a design review isn't "do we log admin actions" — almost every system says yes — it's "can the person whose action is being logged also make that log entry disappear, and if they could, would anyone notice." Tamper-evident is a weaker and more honest claim than tamper-proof: nothing stops a sufficiently privileged attacker from deleting an entire log stream, but a well-designed system (separated storage permissions, hash chaining, periodic external anchoring of the chain's root) ensures that doing so either fails outright or leaves unambiguous evidence behind. Naming that distinction explicitly — what this design prevents versus what it merely makes detectable — is what separates an audit logging story that survives real scrutiny from one that only survives until the first incident review asks who else had delete access to the log bucket.