Architecture/CaseStudies/grab-palana-secure-ai-agent-platform

Grab: Palana — a Secure Kubernetes Platform for Autonomous AI Agents

Problem + constraints

Grab's AI agents moved from IDE plugins and chat assistants into long-running, autonomous workloads — remote dev agents, Slack-triggered task handlers, ops-monitoring bots — with standing access to internal APIs, credentials, source repos, and other services. That shift introduces a risk class that doesn't exist for a human-driven CLI session: an LLM-powered agent's filesystem, logs, process environment, and even its prompt context can all become de facto credential stores if a secret ever passes through them, and a compromised or simply misbehaving agent can't be trusted to police itself or shut itself down cleanly.

Grab needed a way to let engineering teams self-serve agent deployment — spin up an autonomous agent without a security review bottleneck — without losing centralized control over four things: identity, secrets, network egress, and operational visibility. The alternative to building a shared platform was shadow-IT sprawl: every team standing up its own ad hoc agent runtime with its own (likely inconsistent, likely too permissive) handling of credentials and network access.

Solution

Grab built Palana, a Kubernetes-native execution platform every autonomous agent runs on top of, rather than each team hosting its own:

Per-agent namespace isolation. Each agent gets a dedicated Kubernetes namespace, owned by one user plus one agent, with restrictive RBAC, resource quotas, a default-deny network policy, DNS access scoped to only the platform services that agent needs, and a persistent /data volume for its state.
Proxy-only secrets, never handed to the runtime. Vault-backed secrets come in two classes: agent-readable secrets available directly via the agent's service account, and proxy-only secrets, where the agent process only ever sees a placeholder token (e.g. TOKEN_GITHUB_PAT). The real credential never enters the agent's memory, logs, or prompt context — it's substituted downstream.
Enforced egress through Envoy + OPA. All outbound traffic is forced through an Envoy egress proxy running ext-authz checks against Open Policy Agent. The proxy is what swaps a placeholder header for the real Vault-issued credential before forwarding the request upstream — so even a fully compromised agent process never holds a live, exfiltratable token.
Identity-derived, spoof-resistant LLM gateway access. LLM calls route through a proxy wrapper in front of LiteLLM and Grab's internal multi-provider gateway (GrabGPT, itself fronting OpenAI/Azure/Bedrock/VertexAI). The wrapper derives the calling agent's identity from Kubernetes context — not from a client-supplied header, which an agent could forge — then looks up that agent's GrabGPT credential in Vault and routes accordingly. Centralizing this also means an LLM-provider swap is a gateway-side change, not an every-agent-config change.
Externalized kill mechanisms. Because a compromised agent can't be trusted to self-terminate, shutdown lives entirely outside the agent's own runtime: a network-level kill switch that disables an agent's network policies directly from the control plane, and a reaper CronJob that tracks last-observed activity (gateway/proxy logs, Git activity, Slack messages, Prometheus signals) and stops an idle agent after a configurable threshold — while preserving its namespace, RBAC, /data, and Vault state so it can resume later.

Palana currently runs hundreds of production agents at Grab, spanning remote dev environments, Slack automations, and named internal agent workers.

What to steal

Treat "agent identity" as platform-derived, not self-asserted. Deriving identity from Kubernetes context instead of a client header closes the obvious spoofing hole — the same principle as never trusting a client-supplied user ID in a web API.
Proxy-only secrets are the highest-leverage single design choice here. If a credential never touches the workload's process, filesystem, or prompt context, an entire class of leakage vectors (log scraping, prompt injection tricking the agent into echoing a secret, core dumps) is eliminated by construction rather than by hoping the agent behaves.
Design shutdown assuming the workload won't cooperate. Any autonomous system — not just LLM agents — should have a kill path that lives entirely outside the thing being killed. Relying on graceful self-termination is an availability bet you lose exactly when you need it most.
Make the secure path the easy path. Palana's stated goal is self-service deployment without a security review bottleneck — the platform absorbs the security burden so individual teams don't have to reimplement (and inevitably under-implement) it per agent.

Principal Engineer Lens

This is a fresh instance of an old problem — "how do you grant a process just enough access to do its job and nothing more" — applied to a workload class (autonomous LLM agents) whose failure mode is unusually hard to reason about, because the thing that decides what to do next is a model, not deterministic code. That unpredictability is exactly why Grab pushed every hard security guarantee (secret handling, egress control, kill switches) into infrastructure the agent can't influence, rather than trusting agent-level judgment or prompt-level guardrails. In an architecture review, that's the tell of mature thinking: when a component's behavior can't be fully specified or trusted, you don't try to make the component behave — you constrain the blast radius around it. For anyone building or evaluating agentic tooling now (increasingly relevant well beyond Fintech/BigTech, as agent-driven workflows show up everywhere from CI pipelines to internal ops bots), the sharp question is Palana's own framing: does a compromised instance of this thing hold any credential it could exfiltrate, and can you actually turn it off without its cooperation?

Reel Script

Setup: Grab's engineering teams wanted to run autonomous AI agents in production — not chatbots, but long-running workloads with standing access to APIs, credentials, and repos. The problem: an LLM agent's filesystem, logs, and even its prompt context can all become credential leaks, and you can't trust a compromised agent to shut itself down.

Concept walkthrough: Walk through Palana's layers — per-agent Kubernetes namespace with default-deny networking; proxy-only secrets where the agent only ever sees a placeholder token and the real credential is swapped in by an Envoy/OPA egress proxy; an LLM gateway that derives agent identity from Kubernetes context instead of trusting a client header; and kill mechanisms (a control-plane network kill switch, an activity-tracking reaper CronJob) that live entirely outside the agent's own runtime.

Real example / case study tie-in: Trace a single outbound API call from a Palana-hosted agent: it never sees a real credential, only a placeholder — the Envoy proxy is the only place the real token exists, and only for the duration of that one proxied request. Now trace what happens if the agent goes rogue or gets prompt-injected: it still can't produce a usable credential from its own logs or memory, and if it needs to be stopped, the reaper or kill switch does it from outside, with no cooperation required.

Tradeoffs & alternatives: Contrast with letting each team run its own agent sandbox (faster to start, but every team reinvents secret handling and network policy, with predictably uneven results) and with just trusting well-crafted prompts/guardrails to keep an agent in bounds (cheaper to build, but a prompt is not a security boundary — it constrains intent, not capability).