Hermes Wiki
Developer/AI/LLMGateway-EnterpriseRouting/Fundamentals/llm-gateway-pattern

The LLM Gateway Pattern

Concept

Once more than one team in an org is calling LLM APIs, a new failure mode shows up that has nothing to do with model quality: every team ends up building its own auth, its own cost tracking, its own retry/fallback logic, and its own (often absent) redaction of sensitive fields before they leave the org's network boundary. An LLM gateway is a middleware layer that sits between application code and the various model providers — OpenAI, Anthropic, AWS Bedrock, Google Vertex AI — so that every call routes through one place instead of each service importing a different provider SDK directly. The gateway terminates the client request, applies auth/quota/redaction rules, picks (or fails over to) a backend model, and forwards the call; the caller never talks to a provider directly.

This is a server-side, org-wide pattern — distinct from the client-side ProviderAdapter abstraction (see AI/LLMIntegration), which swaps providers at a single call site without touching organizational auth or governance at all. A team can use a ProviderAdapter and still be required to route every one of its calls through a shared gateway; the two patterns solve different problems and commonly coexist.

Concretely, a gateway sitting in front of AWS Bedrock (the open-source bedrock-access-gateway project is a working reference implementation) exposes a single OpenAI-compatible API surface backed by every model Bedrock hosts, while an API-Gateway-fronted deployment adds request authorization, usage quotas/throttling, and WAF-level protection in front of that. The mechanics generalize regardless of which provider sits behind it: unify the API surface, add a policy layer AWS/OpenAI/Anthropic's own SDKs don't provide, and give every downstream caller one thing to authenticate against and one place that sees every request.

Tradeoffs

Approach Benefit Cost
No gateway — every service calls providers directly Zero added latency or infrastructure; simplest for a single team, single provider No unified cost tracking or chargeback; no central kill switch when a provider has an outage; sensitive data can leak to a third-party model with nothing in the path to catch it
Self-hosted OSS gateway (e.g. LiteLLM, bedrock-access-gateway) Full control, traffic can stay inside the org's own network/VPC, no per-request fee to a third party The org now owns an availability-critical service — its own uptime, scaling, and on-call burden
Managed SaaS gateway (e.g. Portkey) Fastest to stand up, built-in observability/caching out of the box Adds a third-party dependency (and often a per-request cost) in the path of every LLM call the org makes
Cloud-native gateway (AWS Bedrock Access Gateway / Amazon API Gateway in front of Bedrock) Traffic never leaves the cloud provider's network; integrates natively with the org's existing IAM/WAF/observability stack Ties the org to that provider's model catalog unless paired with cross-cloud routing on top

A gateway of any kind trades a small amount of added latency (one more network hop, typically single-digit milliseconds for a well-built one — the open-source Bifrost gateway benchmarks around 11 microseconds of overhead per request at 5,000 RPS) for centralized control, visibility, and portability. Below a certain call volume and team count, that trade doesn't pay for itself yet — see "When to use" below.

When to use / when not to

  • Use a gateway as soon as more than one team or use case in an org is calling LLMs — centralizing early avoids retrofitting governance onto a dozen ad hoc integrations later, which is materially harder than building it in from the start.
  • Especially valuable where calls carry sensitive data (device configs, customer PII) that must never leave the org's network unredacted — the gateway is the one place that can enforce that guarantee for every caller, rather than trusting each team to remember.
  • A single team on a single provider, spending a modest amount per month, gets little from a gateway yet — a direct provider SDK call is simpler and the governance problem a gateway solves doesn't exist at that scale.
  • Don't build a bespoke gateway from scratch by default — evaluate an existing OSS or cloud-native option first (LiteLLM, bedrock-access-gateway, Kong's AI Gateway plugins) since the request/response shape, streaming, auth, and rate-limit mechanics for multi-provider LLM traffic are the same problem most teams solve, not something specific to one org.

Common pitfall

Standing up a gateway but leaving an escape hatch: a team that finds the gateway adds friction (or doesn't know it exists) calls a provider directly instead, and now the org has a governance layer that only sees some of its own LLM traffic. The gateway's entire value — unified cost tracking, a central redaction/audit point, one place to kill a misbehaving integration — depends on every call actually going through it. That has to be enforced structurally (network egress rules that block direct provider API endpoints, or credentials that are only ever issued to the gateway itself, never to individual services) rather than by policy alone; a gateway nobody is required to use converges toward being routed around under any real deadline pressure.

Engineering Lens

The LLM gateway pattern is the same "single trusted choke point" idea general API gateways apply to internal-to-external traffic (see Networking/APIGateway), just with LLM-specific concerns layered on: token-based billing instead of per-request billing, provider-specific streaming and error semantics that need normalizing, and redaction rules that have to understand the shape of a prompt rather than a generic HTTP body. The judgment call that actually matters in a design review isn't "we have a gateway" — it's being able to name exactly what the gateway is guaranteed to catch (a leaked API key, an unredacted PII field, an unbounded per-team spend) and showing that guarantee was designed in, not discovered after an incident where a team's direct provider call bypassed it entirely.

Sources

Hermes Wiki