Developer/Auth/SecretManagement/Fundamentals/secrets-management-and-rotation

Secrets Management and Rotation

Concept

A "secret" is any credential that grants access if it leaks — database passwords, API keys, TLS private keys, service-to-service tokens, signing keys. Secrets management is the discipline of never letting those values live in the places they naturally want to end up: hardcoded in source, pasted into a Slack message, sitting in a plaintext .env file committed to a repo, or baked into a container image layer. The standard architecture is a centralized secrets store (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager) that holds the actual values, encrypted at rest, and hands them to applications at runtime through short-lived, audited API calls rather than static files — the application authenticates to the secrets store (often via its own workload identity, not a secret-to-fetch-a-secret bootstrapping problem) and pulls what it needs into memory only.

Rotation is the other half: a secret that never changes has an unbounded blast-radius window — if it leaked six months ago and nobody knows, it's still valid today. Rotation policy forces credentials to expire and be replaced on a schedule (automatically, ideally, not via a manual calendar reminder), which caps how long a leaked credential stays useful even if the leak itself is never detected. The strongest version of this is dynamic secrets: instead of rotating a long-lived shared credential, the secrets store issues a brand-new, uniquely-scoped, short-TTL credential per consumer per session (e.g., Vault generating a fresh database user with a 1-hour lease for each application instance) — nothing long-lived exists to leak in the first place.

Tradeoffs

Approach	Blast radius on leak	Operational overhead	Failure mode if store is down
Hardcoded / static secret, no rotation	Unbounded — valid until manually revoked, if the leak is even noticed	Lowest	N/A — no dependency
Static secret in a vault, manual rotation	Bounded by how disciplined the rotation calendar is (usually not very)	Low	App can't fetch a new value, but cached value keeps working
Static secret, automated rotation	Bounded by rotation interval (hours to days)	Moderate — needs rotation automation + app reload/reconnect logic	Same — cached value survives a short outage
Dynamic, short-TTL, per-consumer secrets	Minutes — leaked value expires almost immediately	Highest — every consumer needs a lease-renewal client and graceful re-auth path	App loses access once its lease expires and the store is unreachable — a new dependency-availability risk

The real tradeoff isn't security-vs-convenience in the abstract, it's security-vs-availability coupling: the more aggressively you rotate, the more your application's uptime now depends on the secrets store being reachable to hand out fresh credentials. Dynamic secrets are the strongest security posture and the strongest new single point of failure, simultaneously.

When to use / when not to

Use a centralized secrets store with automated rotation as the default for anything touching production — database credentials, third-party API keys, internal service tokens.
Push toward dynamic, short-TTL secrets specifically for credentials with high blast radius if leaked (database admin access, payment-processor keys) — the operational cost of lease renewal is worth it there.
A long static secret with manual rotation is a reasonable stopgap for low-blast-radius, low-frequency-use credentials (a rarely-called internal reporting API), not for anything customer-facing or financial.
Don't reach for dynamic secrets everywhere reflexively — for a low-traffic internal tool, the added availability coupling to the secrets store can be a worse risk than the static secret it replaces.
Never accept "it's in an environment variable, not in the code" as sufficient — env vars are still static, unrotated, and visible to anything with process/host access; the goal is short-lived and centrally revocable, not just out of git history.

Common pitfall

Treating "we moved secrets into Vault/Secrets Manager" as the finished state, without ever wiring up rotation. A secrets store with no rotation policy is just a more expensive way to store a static secret — it fixes the "credential in source control" problem but not the "this credential has been valid, unchanged, since day one" problem, which is the property that actually determines how long a leak stays exploitable.

Engineering Lens

The question worth asking in a design review isn't "where are secrets stored" (everyone has an answer to that by now) — it's "what's the maximum time a leaked credential from this system stays valid, and who finds out it leaked." That framing forces the rotation-interval and audit-logging conversation instead of stopping at storage. It also surfaces the availability tradeoff explicitly: adopting dynamic secrets for a payment-processing service is an easy call, but doing the same for every internal cron job adds a hard dependency on the secrets store's uptime that may not be worth the marginal security gain — naming that tradeoff out loud, rather than applying one policy uniformly, is the Principal-level move.