Uber: Zone-Failure-Resilient OpenSearch
Problem + constraints
OpenSearch powers use cases across every tier at Uber, which makes Zone Failure Resilience (ZFR) a non-negotiable requirement rather than a nice-to-have: a deployment has to survive the complete loss of an availability zone — the whole zone, not just one node — without querying or ingestion breaking. OpenSearch ships a built-in feature for this, shard allocation awareness, which tells the cluster to spread shard copies across zones so no single zone holds every copy of any shard. But that feature alone assumes the cluster's view of "which zone is this node in" stays accurate over time, and in a containerized environment it doesn't automatically: nodes get rescheduled, replaced, and rebalanced by the orchestration layer, and nothing inherently guarantees a replacement node lands back in the same failure domain as the node it replaced. Left alone, physical placement can drift out from under the logical awareness configuration, quietly eroding the zone-spread guarantee until an actual zone failure exposes it.
Solution
Uber closed that gap by combining two layers instead of relying on either alone. The physical-placement layer is isolation groups, Uber's own infrastructure construct built on top of Odin (Uber's container orchestration platform): an isolation group is a logical partitioning of nodes across failure domains (zones/racks) that's guaranteed stable over time — a node and its replacement are guaranteed to land in the same isolation group even as the underlying physical host churns underneath. The logical-placement layer is OpenSearch's own shard allocation awareness, configured as forced awareness so the cluster refuses to over-aggressively rebalance shards into the surviving zones the moment one zone goes down — a naive rebalance-everything reaction would overload the survivors right when they're already carrying the failed zone's traffic, extending the outage rather than containing it.
With isolation groups guaranteeing where nodes physically live and forced shard allocation awareness guaranteeing how shard copies are logically spread across those isolation groups, the two layers compose: at least 3 shard copies and 5 cluster-manager nodes spread across isolation groups lets the cluster tolerate a full zone outage plus one additional node failure while still holding quorum and losing zero data. The net effect is that OpenSearch's resilience is decoupled from the churn of physical data-center topology — the orchestration layer can freely reschedule nodes without ever quietly breaking the zone-spread guarantee the cluster depends on.
What to steal
- A platform feature that assumes a static fact ("this node is in zone A") needs a layer underneath it that keeps that fact true over time in a dynamic, container-orchestrated environment — otherwise the guarantee erodes silently and only shows up as a surprise during the actual failure it was meant to survive.
- Separate physical placement (where does this thing actually run) from logical placement (how does the application reason about where things run) as two composable layers, each independently guaranteed, rather than one entangled mechanism. It's much easier to reason about and test each layer in isolation.
- Size quorum-bearing components (cluster managers here) with an explicit failure budget in mind — "zone loss plus one more node" — rather than the bare minimum needed for normal operation. The bare minimum survives the common case; the budgeted number survives the case you're actually building for.
- Guard against the naive "rebalance everything immediately" reaction to a partial outage. The instinct to route around a failure fast can itself cause a second, self-inflicted outage by overloading whatever's left standing — sometimes the resilient move is to hold shape and let the surviving capacity absorb load as-is.
Engineering Lens
This is a strong example of designing for blast-radius containment as a first-class requirement rather than an afterthought bolted onto a working system — "survive a full zone loss, not just a node loss" changes the failure-domain math from the start rather than getting patched in later. It's also a genuinely close mapping to physical network fault-domain design: isolation groups that keep a node and its replacement in the same failure domain despite host churn is the same underlying problem as leaf-spine rack/zone awareness in a physical fabric — directly relevant territory for someone working with Cisco APIC-style fabric design, even though this case study is running in a cloud/container context rather than Uber's own data centers. Beyond the network angle, "how many correlated failures can this system absorb while keeping quorum" is exactly the kind of question a Principal Engineer needs a precise, quantified answer to before an architecture review, not a hand-waved "it's replicated across zones so we're fine."