Source: NVIDIA Technical Blog — 2026-08-11

Summary

NVIDIA released Nemotron 3.5 Lightning, a 30B-parameter mixture-of-experts model with only 3B active parameters per token, purpose-built as the fast, cheap "execution" layer for long-running agents rather than a general chat model. Paired with it is NeMo Switchyard, a router that sends routine execution steps to Lightning and only escalates genuinely hard planning steps to a larger frontier model like Nemotron 3 Ultra. NVIDIA reports Lightning generates around 1,200 tokens per second and outpaces same-class competitors by wide margins on agent-specific benchmarks, while trading away some raw reasoning depth to get there. It's open-weight and runs on hardware from RTX PCs to Jetson.

Key Takeaways

• Architecture: 30B total parameters, only 3B active per token (mixture-of-experts), plus speculative decoding and harness-optimized training aimed specifically at agentic execution rather than open-ended reasoning.
• Speed: NVIDIA cites ~1,200 tokens/second on its own hardware; independent pre-release testing with DeepInfra measured median output closer to ~670 tokens/second — still fast, worth treating NVIDIA's number as a ceiling rather than a typical result.
• Agent-task benchmarks: 86% on PinchBench (multi-step agentic task completion) and 24% on Terminal-Bench v2.1 — more than 3x Nemotron 3 Nano's 7% on the same benchmark.
• General capability benchmarks: MMLU Pro 81.94, SWE-bench Verified 51.56, GPQA Diamond 75.44 (all BF16) — respectable but not frontier-tier, consistent with a model tuned for fast execution over deep reasoning.
• NeMo Switchyard is the actual product idea: instead of running every agent step through one expensive model, route routine execution to Lightning and reserve a bigger model like Nemotron 3 Ultra for the planning steps that need it — a two-tier cost/latency architecture, not just a faster model.
• Ships open-weight with NVFP4 and BF16 checkpoints, deployable on RTX PCs, DGX Spark, GB10 systems, and Jetson, and supported by vLLM and Ollama.

Reel Script

Hook: Most model releases chase bigger benchmarks. Nvidia just shipped one that's deliberately worse at reasoning — because for the parts of an AI agent's job that don't need reasoning, speed is worth more than intelligence.

Core Concept: Long-running AI agents don't spend most of their time thinking hard — they spend it doing routine execution steps: calling a tool, parsing a result, formatting output, moving to the next step in a loop. Running every one of those steps through a large, expensive, slow reasoning model is like hiring a senior architect to fetch coffee. Nemotron 3.5 Lightning is Nvidia's answer to that mismatch: a mixture-of-experts model — think of it as a committee of smaller specialist sub-models where only a few get consulted per question, so it behaves like a 30-billion-parameter model but only spends the compute of a 3-billion-parameter one on each token. That's why it's fast. The companion piece, NeMo Switchyard, is a router sitting in front of it: routine execution steps get sent to Lightning, and only the genuinely hard planning decisions get escalated up to a bigger, slower, smarter model.

Hands-On: The numbers tell the actual story of the tradeoff. On raw speed, Nvidia's own testing clocks Lightning around 1,200 tokens per second, though independent pre-release benchmarking by DeepInfra measured a more modest ~670 tokens/second median — a gap worth remembering any time a vendor cites its own best-case number. On agent-specific tasks, it does genuinely well: 86% on PinchBench, which measures multi-step task completion, and 24% on Terminal-Bench v2.1 — more than triple the 7% scored by Nvidia's own smaller Nemotron 3 Nano. But look at the general-reasoning benchmarks and the tradeoff shows up: MMLU Pro sits at 81.94 and GPQA Diamond at 75.44 — solid, not frontier-class. That's the deliberate design: this model is not trying to out-think GPT-5-class or Claude-class reasoning models. It's trying to be the fast, cheap tier that a router like Switchyard can lean on for 90% of an agent's steps, saving the expensive model for the 10% that actually need it.

Takeaway: If your agent pipeline is paying frontier-model prices for every single tool call and formatting step, a two-tier router architecture — small fast model for execution, big model for planning — is very likely cheaper and faster without meaningfully hurting task success, and Nvidia just open-sourced a reference implementation of exactly that pattern. Worth prototyping before assuming you need the expensive model on every step.

Discussion