Design a Secrets Management System
Scenario prompt
Design a secrets management system for a company running microservices across multiple environments (dev/staging/prod) and multiple cloud accounts. It needs to:
- Let services fetch secrets (DB passwords, third-party API keys, TLS certs) at runtime, with nothing sensitive baked into code, container images, or config files
- Support rotating any secret on a schedule (or on suspected compromise) without requiring every consuming service to be redeployed
- Scope access per-service on a least-privilege basis, with every access fully audited
- Not become a single point of failure — a brief outage of the secrets system shouldn't take down every service that's already running
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — how you'd handle the bootstrap problem (how a service first authenticates to the secrets store), rotation, and the SPOF concern.
Model solution
Centralized secrets store with per-identity policies, not shared static credentials. A dedicated system (HashiCorp Vault, AWS Secrets Manager, or equivalent) holds every secret, encrypted at rest via envelope encryption (a KMS-backed master key wraps per-secret data keys, so the store operator never handles plaintext key material directly). Access is granted per workload identity — an IAM role, a Kubernetes service account, an OIDC-issued identity — never a shared API key baked into an image. This makes "which service can read which secret" a reviewable, revocable policy instead of an implicit fact buried in whoever has the credential file.
Rotation is decoupled from deploys via short-lived credentials and versioning. Rather than a human editing a static password everywhere it's used, the store either issues dynamic, short-lived credentials on demand (a database plugin that mints a scoped, time-boxed DB user per request) or rotates a static secret on a schedule and bumps its version. Client libraries fetch "the latest version" at startup and on a refresh interval, so rotation propagates without a redeploy — the service just picks up the new value on its next fetch/cache-expiry cycle.
Client-side caching with a TTL is what prevents the store from being a SPOF. Services cache the secrets they need in memory with a bounded TTL. If the secrets store has a brief outage, already-running services keep serving on their cached value instead of crashing; only new fetches (cold starts, cache expiry beyond a grace window) are affected. This is a deliberate availability/staleness tradeoff — the same fail-open reasoning as a rate limiter's storage backend — made explicit rather than left as an accident of whatever the client SDK happens to do on timeout.
The bootstrap problem ("secret zero") is solved with platform-native identity, not another secret. A service can't authenticate to the secrets store with a secret it doesn't have yet. The fix is to lean on an identity the platform already vouches for — an AWS IAM instance role, a Kubernetes service account token, an OIDC token from the CI system — and have the secrets store trust that instead of a bootstrapped credential someone has to distribute out-of-band.
Gaps to revisit
- Coordinated rotation for shared secrets. A single DB password consumed by a dozen services needs all of them to pick up the new value inside a bounded window — how do you avoid a partial-rotation state where some services hold the old credential and start failing?
- Detecting and revoking a leaked secret before its scheduled rotation — what monitoring (canary tokens, access-pattern anomaly detection) closes that gap between "credential leaked" and "credential next rotates"?
- Cross-region/cross-account replication of the secrets store itself, and what happens to services in a region that loses connectivity to wherever the store's primary lives.
Engineering Lens
Secrets management is where security and resilience pillars collide directly: the "safe" answer (rotate aggressively, fail closed on any doubt) is in constant tension with the "available" answer (cache generously, fail open), and naming that tradeoff explicitly — rather than defaulting to whatever a library ships with — is exactly the kind of judgment call that separates a working system from a Principal-reviewed one. The bootstrap-trust problem in particular is a good interview signal: candidates who reach for "just another secret" haven't fully internalized that trust has to originate somewhere the platform already vouches for. In Fintech/Capital Markets specifically, secrets management is directly audit-relevant (PCI-DSS, SOC 2 access reviews), so being able to describe the access-audit trail concretely, not just the storage mechanism, reads as production-grade thinking rather than a toy design.