Source: arXiv — 2026-08-03
Summary
RAG-Stack proposes a three-layer system for jointly optimizing RAG serving performance and answer quality, rather than tuning cost and quality as separate, disconnected concerns. It introduces an intermediate representation (RAG-IR) that decouples algorithmic choices from systems choices, a cost model (RAG-CM) that estimates serving performance for a given configuration, and a plan-exploration search (RAG-PE) that searches the combined space for configurations that are both high-quality and cheap to run.
Key Takeaways
- The core insight: RAG pipelines have a huge configuration space spanning both algorithmic choices (which embedding model, which reranker, chunk size) and systems choices (hardware, batching, caching) — and most teams tune these two dimensions independently, missing configurations that are good on both axes at once.
- RAG-IR is an intermediate representation layer that abstracts a RAG pipeline's configuration away from any specific implementation, making it possible to reason about quality and performance for the same underlying pipeline description.
- RAG-CM is a cost model that estimates actual serving performance (latency, throughput, resource cost) for a given RAG-IR configuration without having to fully deploy and benchmark it.
- RAG-PE is the search algorithm that explores the combined quality-performance space using the cost model, aiming to find configurations on the Pareto frontier — the best quality achievable at a given cost, or the lowest cost for a given quality bar.
Reel Script
Hook: Most teams tune their RAG pipeline's accuracy first, then separately try to make it cheaper to run — and end up stuck with a config that's mediocre on both. This system searches quality and cost together.
Core Concept: A production RAG pipeline has a genuinely huge number of dials: which embedding model, how big your text chunks are, whether you rerank, what hardware you serve it on, how you batch requests. The standard workflow is sequential — get the accuracy you want first, then try to optimize the infrastructure around that fixed choice. The problem is that's a local search, not a global one. There might be a completely different pipeline configuration that gets you 95% of the accuracy at a third of the serving cost, but you'll never find it if you only ever explore configurations one dimension at a time. RAG-Stack's answer is to build a system that can reason about both dimensions simultaneously.
Hands-On: The architecture has three layers stacked on top of each other. First, RAG-IR — an intermediate representation — which is just a standardized way to describe a RAG pipeline's configuration, independent of exactly how it's implemented, so the system can compare wildly different setups on equal footing. Second, RAG-CM, a cost model that predicts what a given configuration's actual serving performance would look like — latency, throughput, hardware cost — without having to spin it up and benchmark it live every time, which would be prohibitively slow. Third, RAG-PE, a search algorithm that uses that cost model to explore the space of possible configurations and surface ones sitting on the Pareto frontier — the genuine best-quality-per-dollar options, not just the first config that happened to hit an accuracy target.
Takeaway: If you're running RAG in production and you've only ever optimized quality and cost as two separate passes, this architecture is the case for doing it as one search instead — you're very likely leaving a better configuration on the table. The layered design (representation, cost model, search) is also a clean blueprint worth copying even outside RAG specifically, anywhere you've got a big joint quality-versus-cost space to explore.