Source: MarkTechPost — 2026-07-16
Summary
Moonshot AI announced Kimi K3, a 2.8-trillion-parameter open-weight mixture-of-experts model that activates only 16 of its 896 experts per token. The release pairs a new linear-attention mechanism called Kimi Delta Attention with a training-efficiency technique called Attention Residuals, together claimed to deliver up to 6.3x faster decoding than predecessor K2 at a full 1-million-token context window. Full weights are scheduled to open under a Modified MIT license on July 27; API pricing is already public.
Key Takeaways
- 2.8T total parameters but only 16 of 896 experts (roughly 1.8%) activate per token, keeping per-token inference cost close to a far smaller dense model while retaining a massive total capacity to draw from.
- Kimi Delta Attention is a linear-attention mechanism built specifically to keep long-context decoding fast; Moonshot claims up to 6.3x faster decoding than K2 at the full 1M-token context length.
- Attention Residuals is a separate training-side technique Moonshot says improves training efficiency by roughly 25% for under 2% additional compute cost — distinct from KDA's inference-time gains.
- Open weights land July 27, more than a week after this announcement, so today's release is specs-and-benchmarks-first, with the actual downloadable model following later.
- Published API pricing — $0.30 / $3.00 / $15.00 per million tokens for cache-hit input, cache-miss input, and output respectively — positions K3 as a budget-tier option relative to closed frontier models, despite its scale.
Reel Script
Hook: A new open-weight model just shipped with 2.8 trillion parameters — but for any single word it generates, only sixteen of them actually switch on. That's not a bug, that's the entire trick that makes a model this size affordable to run.
Core Concept: Kimi K3 is a mixture-of-experts model, meaning instead of one giant neural network doing all the work, it's built from 896 smaller specialist networks called experts. For every token the model produces, a router picks just 16 of those 896 experts to actually do the computation — think of it like a hospital with hundreds of specialists on staff, but your specific case only ever gets routed to the handful who are actually relevant. That's how you get a 2.8-trillion-parameter model that doesn't cost 2.8 trillion parameters' worth of compute to run. On top of that architecture, Moonshot added Kimi Delta Attention, a linear-attention mechanism. Standard attention gets slower and more expensive the longer your input gets, because every new token has to compare itself against every previous token. Linear attention compresses that history into a fixed-size running summary instead, so cost stays flat even at huge context lengths — which is exactly why this model can hold a million tokens of context without decoding grinding to a crawl.
Hands-On: The concrete numbers here are what make this more than marketing copy. Only 16 of 896 experts fire per token — under 2% of the model's total parameters are doing any given piece of work. Moonshot claims up to 6.3x faster decoding than their previous K2 model specifically at the 1-million-token context mark, which is where linear attention's advantage over standard attention compounds the most. There's a second, separate optimization here too: a technique called Attention Residuals, which Moonshot says cuts training compute needs by roughly 25% for less than 2% extra cost — that's a training-time efficiency gain, not to be confused with KDA's inference-time speedup. And the pricing is public already: $0.30 per million cached input tokens, $3 per million fresh input tokens, $15 per million output tokens — cheap enough that the economics, not just the specs, are the pitch.
Takeaway: The real story isn't "2.8 trillion parameters" — raw parameter counts are a vanity metric on their own. The story is that mixture-of-experts plus linear attention is how you get frontier-scale capacity without frontier-scale inference bills, and that combination is what's going to decide which open models actually get self-hosted versus which ones just look impressive in a benchmark table. Worth watching once the July 27 weights actually land and independent benchmarks can confirm the 6.3x claim.