Source: NVIDIA Technical Blog — 2026-08-01
Summary
As agentic workloads push context windows into the millions of tokens, attention computation — not the rest of the transformer — increasingly dominates inference latency. NVIDIA's engineering team breaks down exactly how two architecture choices, group size (how many query heads share one KV head) and head dimension, determine GPU throughput during decode, and distills the analysis into four concrete guidelines for model builders who want their architecture to actually run fast on real hardware, not just score well on paper.
Key Takeaways
- Group size (G) — the number of query heads sharing one KV head — is the single biggest lever during decode: raising G from 1 (standard multi-head attention) to 8 delivers roughly an 8x throughput gain by cutting memory traffic and improving GPU compute utilization.
- That gain flattens past G = 16: per-step kernels get so small that fixed setup and post-processing overhead starts to dominate, so there's a real ceiling to how far this trick scales.
- Head dimension should be set to 128 or 256 to align with GPU memory tiles and 128-byte transfer boundaries — mismatched dimensions leave hardware bandwidth on the table regardless of how clever the attention algorithm is.
- Parallelism strategy is dictated by KV-head count: keep tensor parallelism (TP) ≤ KV heads (KH) so every GPU holds a complete KV head; models with very few KV heads (MQA, or GQA with only 2) exhaust that headroom fast and need alternate strategies (attention data/KV parallelism plus expert parallelism for the MoE FFN) instead.
- The core message: attention performance isn't just an inference-engine problem — it's a joint decision between model architects (who set G and head dimension) and infra teams (who pick the parallelism strategy), and getting the pairing wrong silently caps throughput.
Reel Script
Hook (~17s, 38 words): Two AI models can have identical parameter counts and still run at wildly different speeds on the exact same GPU — and NVIDIA just published the specific architecture knob that explains an 8x throughput gap.
Core Concept (~80s, 185 words): Here's the mechanism. In attention, every query token has to compare itself against a set of "key-value" heads to decide what to pay attention to. In old-school multi-head attention, each query head gets its own dedicated KV head — one-to-one, like every customer at a bank getting their own personal teller. That's accurate but wasteful: most of those tellers are barely used per transaction. Grouped-query attention lets multiple query heads share one KV head — like several customers being served by tellers pulled from a shared pool. NVIDIA calls the size of that shared group "G." Push G from 1 up to 8, and you get roughly an 8x speedup during decode, because you're moving far less data in and out of GPU memory for the same amount of useful computation. But this isn't a free lunch forever — past G=16, each individual computation step gets so small that the GPU spends more time on overhead — starting up the next step — than on actual work, and the gains flatten out.
Hands-On (~55s, 130 words): Picture a simple line chart: x-axis is group size G, y-axis is throughput. The curve rockets upward from G=1 to G=8 — that's your 8x — then bends and flattens past G=16. Alongside that, two hard rules NVIDIA gives model builders: set head dimension to 128 or 256, because that's what aligns cleanly with the GPU's memory tile size and transfer boundaries — anything else wastes bandwidth by design, not by bad luck. And for infra teams: keep tensor parallelism less than or equal to your KV head count, so every GPU gets a complete KV head to work with, or you're splitting something that shouldn't be split.
Takeaway (~24s, 52 words): If you're picking or fine-tuning a model for a latency-sensitive agent pipeline, check its group-size and head-dimension configuration before you check its benchmark score — a model architected wrong for its serving hardware will feel slow no matter how good its raw intelligence is. Match the architecture to the metal.