Simplify AI Agent Orchestration with Lakebase Postgres

Source: Databricks Blog — 2026-07-22

Summary

Databricks describes a pattern for orchestrating long-running agentic workloads using nothing but Lakebase Postgres, Databricks Apps, Lakeflow Jobs, MLflow, and Unity Catalog Volumes — explicitly no Kafka, no Redis, and no separate scheduler. The orchestration layer handles coordinating long-running tasks, managing retries, attributing cost, and giving real-time visibility, all backed by a single Postgres database instead of stitching together separate queueing, orchestration, and observability infrastructure. The post uses a document-processing application as the worked example, where the pattern reduced extraction time from hours to minutes.

Key Takeaways

• The core claim: a scale-ready agent orchestrator doesn't need a message queue (Kafka), a cache/lock layer (Redis), and a scheduler as three separate systems — Postgres, used deliberately as a job table plus pub/sub-style polling, can cover all three roles for many agentic workloads.
• Lakebase Postgres plus Databricks Apps together handle queueing, orchestration, and observability, removing the operational overhead of running and reconciling state across multiple specialized infrastructure pieces.
• The orchestration layer's actual job is enumerated concretely: coordinate long-running tasks, manage retries when a step fails, attribute cost per job/tenant, and expose real-time status — a useful checklist for evaluating any agent orchestration design.
• The worked example is a document-processing pipeline combining Lakebase Postgres, Lakeflow Jobs, MLflow, and Unity Catalog Volumes, which cut extraction time from hours to minutes.
• The pitch is fewer moving parts for teams already on the Databricks platform, trading some of the raw throughput ceiling of a dedicated message broker for lower operational complexity — a real tradeoff, not a free lunch.

Reel Script

Hook Most teams building AI agent pipelines reach for Kafka, Redis, and a scheduler before they've processed a single document. Databricks just shipped an architecture that needs none of them.

Core Concept When you're running AI agents that do long tasks — extract data from a document, call a model, retry if it fails, report progress — you need an "orchestration layer": something that tracks which jobs are running, retries the ones that break, and tells you what's happening in real time. The default industry playbook stacks three separate systems for this: a message queue like Kafka to pass work around, a fast cache like Redis to track state and locks, and a scheduler to kick things off on a timer or trigger. Each of those is its own service to deploy, monitor, and keep in sync with the others. Databricks' argument is that a relational database — specifically Postgres, running as their managed Lakebase — can play all three roles at once for a lot of real agentic workloads: a job table is your queue, a status column is your lock, and a query is your dashboard. Think of it like realizing you don't need three separate specialized offices — reception, filing, and the mailroom — when one well-organized front desk can log requests, track their status, and route them, as long as your volume doesn't demand three dedicated departments.

Hands-On The reference architecture: Lakebase Postgres holds the job state (what's queued, what's running, what failed, what needs a retry), Databricks Apps serves as the application layer that reads and writes that state, Lakeflow Jobs handles the actual scheduled/triggered execution, MLflow tracks model calls and runs, and Unity Catalog Volumes stores the files being processed. No separate broker, no separate cache. The concrete before/after: in the document-processing example Databricks walks through, moving to this pattern took extraction time from hours down to minutes — that's the metric to hold onto, since it's the one number in the post you can actually verify against your own pipeline's baseline.

Takeaway This is the right call if you're already on Databricks and your agent workload's throughput doesn't outgrow what a well-indexed Postgres table can handle — it genuinely cuts operational surface area. But don't rip out a working Kafka-based pipeline at real scale just because "no Kafka" sounds appealing; check your actual throughput needs first.

Discussion

(No questions yet — ask follow-ups via a Claude Code chat session on this repo; answers get appended here.)