Hermes Wiki
opportunities/2026-08-22-incremental-citibank-compliance-dq-audit-automation

Incremental Opportunities: Citibank Compliance DQ & Audit Automation (2026-08-22)

Scope: Projects/Citibank-Compliance-DQ-Audit-Automation (created 2026-08-22, status: planning) against Architecture/Challenges/design-an-audit-log-system-for-a-regulated-platform, Architecture/Challenges/design-a-kyc-aml-onboarding-pipeline, Architecture/Fundamentals/change-data-capture, and BlogPosts/2026-08-20-sli-slo-observability-pipeline — all new or changed this window. This project is brand new (created today) and its own "Open Questions" section leaves several design decisions unresolved that this vault's own Architecture material already answers in detail.

Finding 1 — the audit-log challenge's hash-chain + WORM pattern directly answers "what makes the result store provably tamper-evident," a question the project plan doesn't yet ask

The project's Phase 3a Result Store design (dq_runs, dq_failures, rules tables in Postgres, versioned by source_version) currently has no answer for the exact question a real auditor will ask: how do you prove nobody quietly edited a dq_runs row after the fact to make a quarter look cleaner than it was? Architecture/Challenges/design-an-audit-log-system-for-a-regulated-platform names this precisely as a two-part problem — guaranteed emission (solved with a transactional outbox: the DQ result and its audit record commit atomically, then ship async) and guaranteed immutability (solved with hash chaining, not access control alone: each record's hash includes the prior record's hash, so any historical edit breaks the chain forward from that point, closing the "privileged insider quietly edits both the record and the audit log" gap that access control alone can't close).

Concrete next step: extend the dq_runs/dq_failures schema with a prev_hash/record_hash column pair before Phase 3a is built, not after — retrofitting hash-chaining onto an already-running result store means re-deriving history, while building it in from the first row is nearly free. This is exactly the kind of design decision ("I built tamper-evidence in from day one, here's why") that reads as the Principal-level judgment the source Challenge note calls out.

Finding 2 — the KYC/AML challenge's "audit trail as append-only event log, current status as a derived projection" is the missing design principle for the whole result store

The project's own README states the design principle "one generic rules engine, N per-domain YAML configs" but doesn't state an equivalent principle for the result side. Architecture/Challenges/design-a-kyc-aml-onboarding-pipeline's model solution names exactly this: "the current status is a derived projection of the log, never the source of truth itself" — i.e., don't store pass/fail as a mutable status column that gets overwritten on re-run; append every run as an immutable event, and derive "current pass rate for Deposits in Region=US" by querying the log, not by reading a status field that could silently be updated in place. This is the same event-sourcing instinct Architecture/Fundamentals/change-data-capture describes for the dual-write problem more generally — CDC's whole reason for existing is that a "current state" table drifts from the truth the moment two writers touch it without coordination, and the fix is always "make the append-only log the source of truth, derive views from it."

Concrete next step: when designing the dq_runs table, resist adding an is_latest or overwrite-in-place pattern for re-runs of the same rule/domain/region combination — every run should be its own row, with "current status" computed as a query (latest run per rule) rather than stored as mutable state. This also directly de-risks the Phase 3e chat interface's "why did X fail" query, since the full history is queryable by construction rather than needing a separate history table bolted on later.

Finding 3 — OpenLineage (named in the project's own stack) is explicitly the CDC-and-outbox pattern's real-world reference implementation, not a separate concern

The project stack table lists OpenLineage as "the real-world standard to reference/imitate" for lineage, but treats it as its own line item rather than connecting it to the tamper-evidence and event-sourcing patterns above. Architecture/Fundamentals/change-data-capture names the transactional outbox as the mechanism that makes an event stream reliably reflect database state ("write the event to an outbox table in the same DB transaction as the business change, and let CDC stream the outbox — the event and the state change commit atomically") — which is the same mechanism the audit-log Challenge recommends for guaranteed emission, and structurally close to what OpenLineage's own event model expects producers to supply (a lineage event per job run, tied to the actual data version it read/wrote). Treating "tag every run with source_version" (already in the plan) as an outbox-pattern write, rather than a bolt-on column set after the fact, makes the Phase 3b lineage story and the tamper-evidence story (Finding 1) the same mechanism instead of two separate ones.

Finding 4 — the SLI/SLO blog post is a working reference implementation for exactly the Phase 3c layer this project has only sketched

Phase 3c currently lists example SLIs/SLOs (pass_rate >= 99.5%, freshness <= 24h) but no worked mechanism for computing burn rate or firing an alert. BlogPosts/2026-08-20-sli-slo-observability-pipeline is a live-tested, open-source build of exactly this layer — OTel + Prometheus + Sloth (multi-window burn-rate rule generation) + Alertmanager — applied to API latency/error-rate SLOs, but the mechanism (define SLI as a ratio, define SLO as a target over a rolling window, compute burn rate, alert when burning too fast) transfers directly to "pass_rate per domain/region" as the SLI instead of "% requests under 500ms." The post also documents the specific bugs likely to recur here: a silently-inverted good/bat ratio query (the scariest class — looks fine, is wrong) and cold-start NaN on a short rolling window, both plausible failure modes for a freshly-stood-up DQ pass-rate SLO too.

Concrete next step: when building Phase 3c, use Sloth's declarative SLO-spec format (rather than hand-writing PromQL burn-rate queries) even though this project's metric is pass-rate-from-Postgres rather than latency-from-OTel — the burn-rate math (multi-window, multi-burn-rate alerting per Google's SRE workbook) is the same regardless of what emits the underlying SLI. This turns Phase 3c from "figure out SLO alerting from scratch" into "port a validated pattern to a new data source."

Hermes Wiki