NVIDIA Sets a World Record for MoE Pre-Training on GB300 NVL72

Source: NVIDIA Technical Blog — 2026-07-20

Summary

NVIDIA's GB300 NVL72 rack-scale system hit a new pre-training throughput record on DeepSeek-V3 671B, a large Mixture-of-Experts (MoE) model, reaching 1,648 TFLOPs per GPU — roughly 3x the previous Blackwell generation's delivered performance. The gains come from co-designing the full stack: fifth-generation NVLink for intra-rack GPU communication, ConnectX-8 SuperNICs and Quantum-X800 InfiniBand or Spectrum-X Ethernet for scale-out, and software optimizations across Megatron-Core, TorchTitan, and JAX. NVIDIA also reports per-GPU throughput holding above 97% efficiency when scaling from 256 to 1,024 GPUs.

Key Takeaways

• MoE models like DeepSeek-V3 route each token to a small subset of "expert" sub-networks rather than running the whole model, which shifts the bottleneck from raw compute to how fast experts can exchange data across GPUs and racks — that's what NVLink/InfiniBand co-design targets.
• GB300 NVL72 delivers 1,648 TFLOPs per GPU on DeepSeek-V3 671B pre-training, about 3x the earlier delivered performance of the previous Blackwell generation, driven by 1.8 TB/s per-GPU NVLink bandwidth and 130 TB/s non-blocking all-to-all bandwidth within a rack.
• Scale-out relies on NVIDIA ConnectX-8 SuperNICs paired with Quantum-X800 InfiniBand or Spectrum-X Ethernet to keep cross-rack communication from becoming the bottleneck as clusters grow.
• Software-side gains (3x-10x over prior generations) came from Megatron-Core, TorchTitan, and JAX optimizations layered on top of the hardware improvements — not hardware alone.
• Per-GPU throughput stayed above 97% efficiency scaling from 256 to 1,024 GPUs, meaning the added GPUs are doing nearly full useful work rather than getting lost to communication overhead.

Reel Script

Hook Training a trillion-parameter-class model isn't a GPU count problem anymore — it's a networking problem. NVIDIA just proved it by squeezing 3x more throughput per chip out of the same class of hardware, purely through better plumbing.

Core Concept Mixture-of-Experts models like DeepSeek-V3 don't run every parameter for every token — they route each token to a handful of specialized "expert" sub-networks, like a hospital routing a patient to the right specialist instead of making every doctor examine them. That routing means GPUs constantly have to ship data to whichever GPU holds the right expert. So the real bottleneck in training these models isn't how fast one chip can multiply matrices — it's how fast thousands of chips can talk to each other. NVIDIA's GB300 NVL72 attacks that directly: within a rack, GPUs talk over NVLink at 1.8 terabytes per second each, with 130 terabytes per second of total non-blocking bandwidth so any GPU can reach any other GPU without a traffic jam. Across racks, it hands off to ConnectX-8 networking cards and either InfiniBand or Ethernet fabrics built for the same job at cluster scale.

Hands-On The headline number: 1,648 TFLOPs per GPU pre-training DeepSeek-V3's 671-billion-parameter MoE architecture — about 3x the delivered performance of the prior Blackwell generation on the same class of workload. That gain splits roughly into two buckets. Hardware: the NVLink and InfiniBand/Ethernet bandwidth numbers above, which exist specifically to keep expert-routing traffic from stalling the GPUs. Software: NVIDIA reports 3x-10x additional gains layered on top from continuous optimization of Megatron-Core, TorchTitan, and JAX — the training frameworks that decide how work actually gets scheduled onto that hardware. The result that matters most for anyone planning a training run: scaling from 256 to 1,024 GPUs, per-GPU throughput stayed above 97%. That's the number to sanity-check any vendor's scaling claims against — most systems lose real efficiency well before 4x'ing GPU count.

Takeaway If you're evaluating training infrastructure for large MoE models, the interconnect spec matters more than the TFLOPs-per-chip spec on a data sheet — this record was won on networking and software co-design, not raw silicon. Read the full benchmark methodology before you size a cluster off vendor marketing numbers alone.

Discussion

(No questions yet — ask follow-ups via a Claude Code chat session on this repo; answers get appended here.)