Design a Feature Flag / Progressive Delivery Platform
Scenario prompt
Design a feature flag system for an organization with hundreds of engineers across many teams, shipping to a mix of web, mobile, and backend services. It needs to:
- Let any team gate a new code path behind a flag, targeted by user segment, percentage rollout, or environment, without a deploy to change the flag's state
- Evaluate flags with very low latency and without a network call on every request — flag checks happen in hot paths, sometimes thousands of times per second
- Keep flag state consistent enough that a user doesn't see a feature flicker on and off across requests or see inconsistent behavior across web and mobile in the same session
- Prevent the flag system itself from becoming a liability — stale flags nobody cleans up, or a flag change that silently breaks a service that assumed a flag was permanent
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — how you'd get flag state to every service without a hot-path network call, how you'd keep evaluation consistent for a given user, and how you'd stop flag debt from accumulating.
Model solution
Evaluation happens locally against a cached ruleset, not via a network call per check. A central control plane owns flag definitions and targeting rules (which segments, what percentage, which environments), but every consuming service holds a local, periodically-refreshed (or streamed-via-SSE/webhook) copy of the ruleset and evaluates flags in-process. This is the same shape as the secrets-management client-side-cache pattern: the control plane is the source of truth, but the hot path never blocks on reaching it. A flag check becomes a hash-and-compare against local data — cheap enough to call thousands of times per second.
Consistent evaluation per user comes from deterministic, stable hashing — not random assignment per request. Percentage rollouts and segment targeting are computed by hashing a stable identifier (user ID, device ID, or session ID) combined with the flag's own key, then comparing the hash against the rollout percentage. The same user always lands on the same side of the same flag as long as the ruleset hasn't changed, which is what prevents mid-session flicker — it's not that the value is "sticky" in a stateful sense, it's that the computation is deterministic given the same inputs. Cross-platform (web/mobile) consistency in the same session requires the same identifier and the same targeting rules to be evaluated the same way on both — usually solved by having the client SDK fetch a pre-evaluated bundle for that identifier from an edge service once per session, rather than each platform independently reimplementing the hashing logic.
Propagation lag is a bounded, understood tradeoff, not an assumed-instant guarantee. Pushing a new ruleset to every service instance takes some real time (a polling interval, or the fan-out latency of a push/streaming mechanism) — during that window, different instances may serve different flag states to the same user (e.g., across a load-balanced request happening to hit two different service instances). This should be sized and communicated explicitly ("propagation is typically under 10 seconds"), not left implicit, since it directly answers "why did I see the old behavior for a moment after we flipped the flag."
Flag debt is a lifecycle problem the platform has to design for, not a hygiene problem left to individual teams. Every flag should be created with an owner, a purpose (release gate, kill switch, experiment, permanent config), and — for release-gate and experiment flags specifically — an expected removal date. The platform should surface (and ideally alert on) flags that are fully rolled out or fully off for an extended period and still gated in code, since those are pure tech debt: dead branches nobody has cleaned up, and a hidden source of "the flag system caused an outage" incidents when someone assumes a flag is permanent and it gets removed, or assumes it's temporary and it silently becomes permanent.
Gaps to revisit
- How do you handle a flag whose evaluation depends on data the client doesn't have yet (e.g., a targeting rule based on a user attribute computed server-side) without reintroducing a network call on the hot path?
- What's the blast-radius containment story if a bad ruleset push (a typo'd 100% rollout instead of 1%) reaches every service at once — is there a kill switch for the flag system's own propagation, separate from individual flags?
- Multi-team ownership: what happens when two teams' flags interact (feature A assumes feature B's flag is off) — does the platform have any way to surface that dependency, or is it purely a code-review-time responsibility?
Engineering Lens
Feature flagging looks like a small utility problem but is actually an organizational-complexity problem wearing a technical costume: the hard part isn't "store a boolean somewhere," it's designing a system that stays coherent when hundreds of engineers across many teams are independently creating, targeting, and (hopefully) retiring flags without central coordination. The lifecycle-ownership design (owner, purpose, expected removal date) is the detail that separates a toy flag library from a platform that scales past one team — and calling that out unprompted in a design review is a strong Principal-level signal, since it shows you're designing for the organization's failure modes, not just the request-path's. The deterministic-hashing mechanism itself is a reusable pattern worth recognizing elsewhere — consistent hashing shows up again in sharding and load balancing, so a candidate who names the connection is demonstrating pattern transfer, not just recall.