Networking for SI, Part 2: The Four Networks in an SI Cluster

An SI training cluster runs on several networks at once. NVIDIA’s reference architecture for the DGX SuperPOD with GB200 runs five network fabrics on four physical ones, and Meta runs its training clusters on two independent networks. In both designs, storage traffic stays off the fabric the GPUs use to talk to each other.

Part 1 covered NVMe over Fabrics (NVMe-oF), which carries storage commands across a network. This part asks what each network around it carries, why storage is kept apart from GPU traffic, and how much bandwidth storage needs. For the last question it works from the checkpoint workload in MLPerf Storage. That MLCommons benchmark times how fast a storage system saves and restores a training checkpoint.

Four networks, one cluster

Vendors name these networks differently, so this guide uses four plain names. Scale-up joins GPUs inside one domain. Back-end scale-out joins GPUs across servers and racks, storage reaches shared storage, and the front-end carries what users and administrators touch. Meta’s back-end runs RoCE, which carries remote direct memory access (RDMA) over Ethernet.

Sources: NVIDIA DGX SuperPOD reference architecture with GB200, chapters 3 and 4, Meta’s post on RoCE networks for distributed AI training at scale, and the Llama 3 paper, section 3.3.1.

NVIDIA also runs a fifth network for out-of-band management. It reaches the baseboard management controller (BMC) and management port of every server, switch, storage system, and rack power distribution unit. In its B200 reference architecture, NVIDIA keeps it apart because no user ever needs those ports, and it carries no training or storage data.

Each GB200 compute tray holds two GB200 Superchips, which makes four B200 GPUs and two Grace CPUs. Four ConnectX-7 network interface cards (NICs) run InfiniBand NDR at 400Gb/s for the compute network, and two BlueField-3 cards supply 2×200Gb/s for the in-band and storage networks. NVLink leaves the tray through blind-mate connectors at the back of the rack and runs only to the switch trays in that rack. Every other network leaves through cards and ports on the tray itself.

Scale-up stays inside the rack

In a DGX GB200 NVL72 rack, all 72 GPUs share one NVLink domain. Inside it are 18 compute trays and nine NVLink switch trays with two switch chips each. Every B200 GPU has 18 NVLink 5 links and runs one to each of the 18 switch chips, for 1.8TB/s per GPU.

NVIDIA calls this full-mesh connectivity. Because every GPU has a link to every switch chip, this guide reads it as any GPU reaching any other in the rack through one switch chip. NVIDIA’s parts table describes the NVLink switches as giving direct memory access between GPUs on the same rack, which suggests one GPU can reach another’s memory without a NIC in the path. That memory is HBMHigh Bandwidth Memory stacks DRAM dies on a base die and links them with thousands of vertical wires, giving a very wide, short path to a processor on the same package., which Working Memory Part 4 covers.

Not every vendor uses scale-up this way. Meta’s post on its RoCE networks uses it for the Ethernet switch that joins GPUs within one rack, below a spine tier it calls scale-out. In this guide’s terms, Meta’s scale-up domain is the NVLink inside each eight-GPU server, and that rack switch belongs to its back-end. In NVIDIA’s design the scale-up domain is NVLink too, and the storage ports sit on the BlueField-3 cards instead.

The back-end fabric between racks

GPU-to-GPU traffic between racks runs on the back-end network, which NVIDIA calls the compute fabric. For GB200 that network is InfiniBand NDR, which peaks at 400Gb/s per direction, and it is rail-aligned. A tray has four rails, so each of its GPUs has its own. A scalable unit (SU) of eight racks and 576 GPUs gives every rail its own group of switches, with eight leaf switches, one per rack, under six spine switches.

Within a rack, the trays reach each other on a rail through that rack’s leaf switch. NVIDIA routes traffic between different racks, or between different rails, through the spine layer. A message from the first GPU of a tray in one rack to the first GPU of a tray in another rack climbs from one leaf switch to a spine and back down to the other rack’s leaf.

Meta runs both kinds. According to its Llama 3 paper, Llama 3 405B trained on a RoCE fabric and the smaller Llama 3 models on InfiniBand, both with 400Gb/s links between GPUs. Meta’s network post describes the back-end as a non-blocking fabric between every RDMA NIC, and the paper qualifies that for its 24,000-GPU RoCE cluster. Pods of 3,072 GPUs have full bisection bandwidth, the aggregation layer above the pods is oversubscribed 1:7, and Meta’s scheduler and model parallelism keep communication inside a pod.

Storage and front-end links

NVIDIA’s B200 reference architecture sets out the storage fabric’s job plainly. It is independent of the compute fabric, gives each DGX B200 at least 40GB/s, and runs over InfiniBand or RoCE. Storage systems connect at a 1:1 port-to-uplink ratio. On the server side, DGX B200 connections are oversubscribed near 4:3, so the server-facing ports add up to about a third more than their switches’ uplinks.

GB200 trays get a storage line rate of 2×200Gb/s each, again over RoCE. The front-end, which NVIDIA calls the in-band management network, handles node provisioning, data movement, Internet access, and the services users reach, at 200Gb/s bonded. NVIDIA also expects a Network File System (NFS) share on that network for home directories, scratch space, and logs, and sets 100Gb/s as its minimum.

Divided per GPU, the storage share is the smallest. A tray’s four GPUs split 2×200Gb/s of storage line rate, so each gets 100Gb/s. Each GPU has its own 400Gb/s compute link and 1.8TB/s of NVLink. NVLink’s figure is a total and InfiniBand’s is per direction, so the comparison mixes bases. On those numbers, NVLink is 36 times the compute share, and the compute share is four times the storage share.

Each tray also carries four 3.84TB E1.S drives in a RAID0 set, which NVIDIA lets users put to work caching or staging data. A dataset cached there on its first read doesn’t have to cross the storage network again.

Why storage stays off the GPU fabric

In the B200 design, NVIDIA keeps storage on a fabric independent of the compute fabric to maximize performance of both storage and application work. With GB200 it kept that split and merged storage with the front-end instead. Storage and in-band management now share one Ethernet fabric. NVIDIA made that change for cost-efficiency. Storage runs on its own Virtual Extensible LAN (VXLAN) segment and takes two of the four ports on each BlueField-3 card.

Each SU carries 16×800Gb/s of non-blocking bandwidth to the storage systems, while the server side blocks at 5:3. RoCE is a basic requirement for those storage systems, and NVIDIA relies on congestion control and adaptive routing in the switches to hold latency down.

Meta draws the line in the same place with fewer networks. Its front-end carries data ingestion, checkpointing, and logging, and the back-end carries only training traffic between RDMA NICs. Meta tunes the two differently. Meta calls Data Center Quantized Congestion Notification (DCQCN) the gold standard for storage-focused networks, yet it ran its 400G training back-end for over a year on Priority-based Flow Control (PFC) alone, with DCQCN turned off.

Checkpoints are the traffic that makes the split worth having. According to the Llama 3 paper, checkpoint writes are highly bursty and saturate the storage fabric for short durations. On a shared fabric, that burst would queue in the same switch buffers as the training job’s GPU-to-GPU exchanges, and Part 1 covers what happens when many senders hit one receiver. Both designs put the boundary between GPU-to-GPU traffic and everything else, and this guide reads that as the boundary that matters. A storage network can share a fabric with management traffic, but in neither design does it share one with the training job’s own collectives.

Sizing the storage network from a checkpoint

MLCommons added a checkpoint workload to MLPerf Storage in version 2.0, announced August 4, 2025. It builds the model and optimizer state of a Llama 3 model in memory, saves and loads it with PyTorch, and times the storage system underneath. That makes it a measured stand-in for the burst the storage network has to carry, and Part 1 showed how long a job can wait on that path when storage bandwidth is short.

What a checkpoint holds

The benchmark fixes four model sizes, each with a set process count and checkpoint size. Table 2 of the rules labels the sizes GB and TB but counts them in binary units, so MLCommons’ results page gives them in GiB.

Sources: MLPerf Storage benchmark results page, checkpointing workload table, and MLPerf Storage Benchmark Rules, Table 2.

Most of that is optimizer state. MLCommons’ workload post says the 8B checkpoint totals 105GB, of which 90GB is optimizer state, and that model and optimizer state stay the same size as more data-parallel copies are added. By that rule, a 405B checkpoint stays about 5,290GiB whether 512 GPUs train the model or 16,000 do.

The 405B model runs with tensor, pipeline, and data parallelism of 8×32×2, so its 512 processes each write about 10.3GiB. That sits inside the 9 to 20GiB per process given on the results page. Every write ends with an fsync. A run saves 10 checkpoints before it reads 10 back, and each client host is expected to write more than three times its memory, so the client’s RAM can’t hide the storage.

How long training can wait

Synchronous training stops while a checkpoint is written, and NVIDIA’s GB200 guide says the same of its own clusters. Failures decide how often a checkpoint is needed. During a 54-day stretch of Llama 3 405B pre-training on up to 16,000 H100 GPUs, Meta counted 419 unexpected interruptions. MLCommons built a failure model on that data, and it predicts 7.67 failures a day at 16,000 accelerators against the 7.76 observed.

MLCommons then applies a rule of thumb from its checkpoint workload post. Losing 5% or less of progress is reasonable, which means about 20 checkpoints between expected failures. At 16,000 accelerators it estimates 155 checkpoints a day. That is one every 9.3 minutes, and keeping the overhead under 5% leaves 28 seconds to write each one. At 100,000 accelerators it is 967 a day, every 1.5 minutes, with 4.4 seconds per checkpoint.

The write rate that follows

Storage must absorb the checkpoint size divided by the time allowed:

write rate = checkpoint size / time allowed

Checkpoint size is in GB, time allowed is in seconds, and the write rate comes out in GB/s. Network rates are decimal, so the 405B checkpoint’s 5,290GiB becomes about 5,680GB here. Inside the 28-second budget that is 5,680GB / 28 seconds, or about 203GB/s. At 4.4 seconds it is about 1,291GB/s, roughly 1.3TB/s. Both are MLCommons’ budgets applied to MLPerf’s 405B checkpoint, so they are estimates, not any cluster’s measurement. The second pairs that checkpoint with the budget for a far larger cluster, so it shows scale rather than a real job.

For scale, Meta’s Llama 3 storage fabric, built on its Tectonic file system, offered 240PB on 7,500 SSD servers with 2TB/s sustained and 7TB/s peak throughput. At the sustained rate, 5,680GB would take about 2.8 seconds. That divides one source’s throughput by another’s checkpoint size, so it is arithmetic, not a measured save.

Per GPU, the numbers are small. Spread across 16,000 GPUs, 5,680GB is about 355MB each, inside the 1MB to 4GB per GPU that the Llama 3 paper reports, and writing it in 28 seconds takes about 13MB/s per GPU. At this scale the limit is the aggregate rate the storage end can absorb when every writer starts at once, not the per-GPU link. A save in MLPerf is done only when every process is done, so the slowest writer sets the finish.

One scalable unit, worked through

The 405B workload’s 512 processes fit inside one GB200 SU of 576 GPUs, which makes NVIDIA’s own numbers a fair check. Its 16 storage-side links of 800Gb/s add up to 1,600GB/s. NVIDIA’s storage guidelines for one SU are far lower, with 20GB/s of aggregate write at its standard level and 62GB/s at enhanced, against 40GB/s and 125GB/s of read. NVIDIA also advises write performance of at least half the read performance for large models.

Writing 5,680GB takes about 3.6 seconds at the SU’s 1,600GB/s storage-side line rate, about 92 seconds at the enhanced 62GB/s, and about 284 seconds at the standard 20GB/s. That storage-side capacity is about 26 times the enhanced write rate NVIDIA plans the storage system to deliver. In this design the stall is set by how fast the storage system completes durable writes, which is what MLPerf’s fsync rule and its write duration, the time training would stall, measure.

Sizing the storage network from link rates alone would promise checkpoints far faster than the storage behind it can finish them. The better sizing input is a published 405B write duration for the storage system being bought, compared only with other 405B rows, as MLCommons advises.

What to watch

Faster checkpoints change this arithmetic. On its GenAI training clusters, Meta has optimized checkpointing across thousands of ranks to run in hundreds of milliseconds, a pattern the synchronous, fsync-bound save that MLPerf times leaves out. MLPerf’s subset mode tests another shortcut. It puts the first checkpoint tier on an NVMe drive in each node and uses 8 processes. MLCommons notes that subset mode doesn’t give full fault tolerance without a secondary storage solution or extra software, so a copy still has to reach shared storage.

The rules are moving too. From edition 4.0, MLPerf Storage will compare checkpoint results only within one emulated accelerator, so NVIDIA B200 and AMD MI355 rows will stop lining up. Until then, a 405B write duration from the published results is the closest public stand-in for the stall a storage system adds.

Racks and Rows tracks the SI clusters being built and the storage behind them, and Photon Fabric follows the optics for memory and storage links. In a GB200 tray, the storage and front-end ports sit on BlueField-3 cards, and Part 3 looks at what those cards do beyond carrying packets.