Design a Secrets Management System
Scenario prompt
Design a secrets management system for a company running microservices across multiple environments (dev/staging/prod) and multiple cloud accounts. It needs to:
- Let services fetch secrets (DB passwords, third-party API keys, TLS certs) at runtime, with nothing sensitive baked into code, container images, or config files
- Support rotating any secret on a schedule (or on suspected compromise) without requiring every consuming service to be redeployed
- Scope access per-service on a least-privilege basis, with every access fully audited
- Not become a single point of failure — a brief outage of the secrets system shouldn't take down every service that's already running
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — how you'd handle the bootstrap problem (how a service first authenticates to the secrets store), rotation, and the SPOF concern.
Model solution
Centralized secrets store with per-identity policies, not shared static credentials. A dedicated system (HashiCorp Vault, AWS Secrets Manager, or equivalent) holds every secret, encrypted at rest via envelope encryption (a KMS-backed master key wraps per-secret data keys, so the store operator never handles plaintext key material directly). Access is granted per workload identity — an IAM role, a Kubernetes service account, an OIDC-issued identity — never a shared API key baked into an image. This makes "which service can read which secret" a reviewable, revocable policy instead of an implicit fact buried in whoever has the credential file.
Rotation is decoupled from deploys via short-lived credentials and versioning. Rather than a human editing a static password everywhere it's used, the store either issues dynamic, short-lived credentials on demand (a database plugin that mints a scoped, time-boxed DB user per request) or rotates a static secret on a schedule and bumps its version. Client libraries fetch "the latest version" at startup and on a refresh interval, so rotation propagates without a redeploy — the service just picks up the new value on its next fetch/cache-expiry cycle.
Client-side caching with a TTL is what prevents the store from being a SPOF. Services cache the secrets they need in memory with a bounded TTL. If the secrets store has a brief outage, already-running services keep serving on their cached value instead of crashing; only new fetches (cold starts, cache expiry beyond a grace window) are affected. This is a deliberate availability/staleness tradeoff — the same fail-open reasoning as a rate limiter's storage backend — made explicit rather than left as an accident of whatever the client SDK happens to do on timeout.
The bootstrap problem ("secret zero") is solved with platform-native identity, not another secret. A service can't authenticate to the secrets store with a secret it doesn't have yet. The fix is to lean on an identity the platform already vouches for — an AWS IAM instance role, a Kubernetes service account token, an OIDC token from the CI system — and have the secrets store trust that instead of a bootstrapped credential someone has to distribute out-of-band.
Gaps to revisit
- Coordinated rotation for shared secrets. A single DB password consumed by a dozen services needs all of them to pick up the new value inside a bounded window — how do you avoid a partial-rotation state where some services hold the old credential and start failing?
- Detecting and revoking a leaked secret before its scheduled rotation — what monitoring (canary tokens, access-pattern anomaly detection) closes that gap between "credential leaked" and "credential next rotates"?
- Cross-region/cross-account replication of the secrets store itself, and what happens to services in a region that loses connectivity to wherever the store's primary lives.
Principal Engineer Lens
Secrets management is where security and resilience pillars collide directly: the "safe" answer (rotate aggressively, fail closed on any doubt) is in constant tension with the "available" answer (cache generously, fail open), and naming that tradeoff explicitly — rather than defaulting to whatever a library ships with — is exactly the kind of judgment call that separates a working system from a Principal-reviewed one. The bootstrap-trust problem in particular is a good interview signal: candidates who reach for "just another secret" haven't fully internalized that trust has to originate somewhere the platform already vouches for. In Fintech/Capital Markets specifically, secrets management is directly audit-relevant (PCI-DSS, SOC 2 access reviews), so being able to describe the access-audit trail concretely, not just the storage mechanism, reads as production-grade thinking rather than a toy design.
Reel Script
Setup: Every service needs passwords, API keys, and certificates to talk to the outside world — where do those live, and what happens the day one of them leaks?
Concept walkthrough: Walk through the shift from "secrets in a config file" to "secrets fetched at runtime from a central store" — and why that alone doesn't solve rotation. Explain dynamic short-lived credentials vs. rotating a static secret and versioning it, and how a client-side cache with a TTL is what lets rotation happen without a redeploy.
Real example tie-in: Walk through the bootstrap problem concretely: a brand-new service instance starts up with nothing — how does it prove who it is to the secrets store without already holding a secret? Show how platform-native identity (an IAM role, a service account token) breaks that circularity.
Tradeoffs & alternatives: Contrast fail-open (serve stale cached secrets during an outage, risk using a soon-to-be-revoked credential) vs. fail-closed (refuse to serve without a fresh fetch, risk a full outage cascading from a secrets-store blip) — same shape of tradeoff as a rate limiter's storage failure mode, just with security stakes instead of availability stakes.
Principal Engineer takeaway: The interesting part of this design isn't "where do secrets live" — it's the bootstrap-trust problem and the fail-open/fail-closed call under outage, both of which force you to be explicit about where trust originates and what you're willing to risk when the store itself is unavailable. That explicitness is what audit reviewers and design reviewers are both actually looking for.