Hermes Wiki
Synthesis/ai-gateway-is-api-gateway-pattern-reapplied

Synthesis: Four Vendors Ship the Same 'AI Gateway' in One Week — It's the API Gateway Pattern, Just Pointed at Models and MCP Servers Instead of REST Endpoints

The connection

Within roughly one week, four vendors independently shipped a structurally identical product: a single gateway resource that fronts multiple AI providers/models/MCP servers behind one endpoint, with centralized policy, rate limiting, and credential management — the same job an API gateway or service mesh ingress does for REST/gRPC traffic, just re-scoped to "models, MCP servers, and tools" as first-class objects instead of "APIs and operations":

Why this wasn't visible before

Each of these was covered individually in AIDigest as a standalone vendor-feature announcement, with no note connecting them to each other or back to the architecture vault's own gateway-adjacent vocabulary. But Architecture/Fundamentals/ already has the exact concept notes this pattern is built from — load balancing (round-robin now possible specifically because MCP dropped sticky sessions) and service mesh (a mesh's ingress/policy-enforcement-point role is precisely what these four products replicate for AI traffic). None of the four AIDigest entries reference either fundamentals note, and neither fundamentals note was written with an AI-gateway example in mind — the connection existed structurally but had never been named.

The stateless MCP spec change is also the mechanism, not just a coincidence: sticky sessions and a shared session store were the blocker to plain load-balanced gateway fronting; removing them is what makes "one gateway, any backend instance" viable the same way it's viable for stateless HTTP services behind a standard load balancer.

What this suggests

  • This is the same "gateway as policy-enforcement chokepoint" idea as Synthesis/agent-enforcement-moves-outside-the-reasoning-loop (Dogwood, Agent Baseline, Palana) — but applied to routing/cost/rate-limiting concerns instead of security. Both syntheses point at the same underlying shift: agentic infrastructure is converging on "put a gateway in front of everything the agent can call," for both safety and operational reasons.
  • Azure's flagged gap (one key authorizes the whole gateway) is a concrete instance of the credential-scoping problem the credential-scoping skill draft (unresolved) already names — worth citing as a live example next time that draft is revisited.
  • Envoy, Consul, and HAProxy (all newly added Tools notes) are the non-AI-specific ancestors of exactly this pattern — a good "compare the old and new gateway" teaching pair for the explainer-video project if a "what is an AI gateway, really" episode is ever in scope.
  • Projects/RBC_Work's network chatops MCP toolbox (Grafana MCP, Elasticsearch MCP, network-cli MCP) is currently federated by hand; this vendor convergence suggests the "one gateway fronting many MCP servers with centralized policy" shape is becoming a standard component worth evaluating rather than continuing to hand-roll.
Hermes Wiki