Source: arXiv — 2026-07-20 (accepted for ISSTA 2026)
Summary
A paper accepted at ISSTA 2026 (Kim, Pasini, Tonella) proposes Chunk Coverage, an oracle-independent test-adequacy criterion for the retrieval component of RAG systems, borrowed directly from software testing's code-coverage tradition: it measures the fraction of a corpus's chunks that get retrieved at least once across a test suite. The pitch is a reframe — retrieval quality has mostly been treated as an information-retrieval metrics problem (precision/recall on a fixed set of queries), but this paper argues teams also need to know whether their test suite is exercising the retriever broadly across the corpus, or just repeatedly hitting the same popular chunks.
Key Takeaways
- Chunk Coverage (CC) = the number of distinct corpus chunks retrieved at least once by a test suite, divided by the total number of chunks in the corpus — a direct import of the code-coverage concept ("what fraction of my code did my tests actually run") into RAG retrieval testing.
- It's explicitly framed as oracle-independent — you don't need to know the "correct" answer to compute it, only which chunks got retrieved, which makes it cheap to compute continuously (e.g., in CI) even without hand-labeled ground truth for every query.
- The motivating gap: most RAG evaluation focuses on IR-style metrics (precision@k, recall, NDCG) over a fixed query/answer set, which says nothing about whether huge swaths of the indexed corpus are ever exercised by tests at all — chunks that never get retrieved are effectively "dead code."
- Accepted at ISSTA 2026 (the International Symposium on Software Testing and Analysis), signaling that RAG retrieval testing is starting to get treated with the same formal rigor as traditional software test-adequacy research, not just IR benchmarking.
- The authors released an implementation and reproduction artifact on GitHub, making it a practical, directly reusable tool rather than a purely theoretical proposal.
Reel Script
Hook (~16s, 38 words) Your app probably has a code-coverage number — some percent of your codebase your tests actually run. Now ask: what percent of your RAG system's knowledge base do your tests actually retrieve from? Most teams have never measured it. It's usually zero.
Core Concept (~55s, 120 words) In normal software testing, coverage tells you what fraction of your code your test suite actually exercises — untested, "dead" code shows up immediately. RAG systems have never had an equivalent. Teams eyeball a handful of question-answer pairs, check if the retriever found roughly the right passage, and call the system validated, but nobody's checking whether that test suite reaches 5% of the indexed corpus or 95%. A new paper, accepted at ISSTA — software testing's top academic venue — imports the coverage idea directly into retrieval: it defines Chunk Coverage as the fraction of corpus chunks that get retrieved by at least one test query.
Hands-On (~55s, 120 words) The mechanic is simple enough to build yourself: run your test suite, log every chunk ID your retriever returns, then divide distinct chunks hit by total chunks in your corpus. Chunks that never show up are your RAG system's dead code — could be genuinely irrelevant filler, or could be a chunk that's structurally unreachable because of a bad split, a bad embedding, or a chunking boundary that mangled it, and you won't find out until a real user's query needs exactly that passage. The authors published their implementation and data on GitHub rather than keeping it as a paper-only idea, so it's directly reusable against your own pipeline.
Takeaway (~22s, 50 words) If you ship a RAG pipeline and only track precision and recall on a fixed test set, you're validated on the queries you thought to write, not the corpus you actually indexed. Chunk coverage is cheap to instrument — log retrieved chunk IDs, compute the ratio. Add it to CI.