Networking for SI, Part 1: NVMe-oF and the Network Behind SI Storage

An NVMe SSD usually sits on a PCIe bus a few inches from the processor. NVMe over Fabrics (NVMe-oF) sends the same commands across a network, so a server can read a drive that lives in another box. In an SI cluster, that network carries checkpoints, training data and cached context, the three jobs this part follows after the protocol itself.

SPDK’s published NVMe-oF tests against a null device, with no SSD behind it, put the network and software cost at 4.72µs to 21.39µs per read, and those numbers hold only for that setup. The SI training and serving papers below report their delays in storage bandwidth, data preparation and congestion instead. Flash Foundations Part 1 introduces NVMe-oF in a few paragraphs, and the protocol detail continues below.

Same commands, different wire

Local NVMe is built on paired submission and completion queues. According to the NVMe Base Specification, Revision 2.4, a drive can have up to 65,535 I/O queues with up to 65,535 outstanding commands in each, and every command is 64 bytes. Over PCIe, those queues live in host memory. Software tells the drive about a new command by writing to a doorbell register. Part 1 explains why NVMe replaced AHCI with this design.

A network has no shared memory to write into, so NVMe-oF wraps each command in a capsule. A command capsule carries the 64-byte command and can carry data with it. On PCIe a command counts as submitted once the doorbell write moves the queue tail past it. Over a fabric, the base specification counts it as submitted “when a host adds a capsule to a Submission Queue”.

NVM Express’s own NVMe over Fabrics overview says the fabric version keeps “the entire NVMe multi-queue model”, with normal submission and completion queues “encapsulated over a message-based transport.” The same overview says roughly 90% of the protocol matches local NVMe, which is the consortium describing its design rather than a measurement.

Three ways to carry a capsule

NVMe-oF hands the network to a transport binding, and three bindings exist. Two have NVM Express specifications of their own, and the third lives in a Fibre Channel standard.

RDMA stands for remote direct memory access. NVM Express’s binding for it is the NVMe over RDMA Transport Specification, Revision 1.3. The consortium’s overview counts “any of the RDMA technologies, including InfiniBand, RoCE and iWARP” as usable for NVMe over Fabrics. RoCE is RDMA over Converged Ethernet.

TCP needs no special network cards. The NVMe over TCP Transport Specification, Revision 1.3 is written to allow “efficient software-only implementations” on ordinary TCP stacks while leaving room for hardware offload. NVMe-oF traffic uses TCP port 4420, and discovery uses port 8009. Both ports are registered to NVM Express in the IANA port registry, as nvme and nvme-disc.

Fibre Channel is the third option, and Base 2.4 places its binding in the INCITS 556-2020 standard (FC-NVMe-2), which ANSI sells. Adapters for it are on sale. Broadcom’s Emulex Gen 7 HBAs support NVMe over Fibre Channel, and Marvell’s QLogic 2870 Series runs FCP and FC-NVMe concurrently on the same port.

Ethernet products cover both RDMA and TCP. NVIDIA’s ConnectX-7 datasheet says the adapter “accelerates NVMe-oF over both RoCE and TCP”. Western Digital’s OpenFlex Data24 4000 storage platform uses RapidFlex A2000 fabric bridges with 12 ports of 100GbE that connect to RDMA or TCP hosts. KIOXIA’s EM6 connects over single or dual 25Gb Ethernet with RoCEv2. KIOXIA’s KumoScale target software supports both transports, though its page says the product is now maintenance only.

Finding drives, and more than one path to each

Every host and storage subsystem on the fabric has an NVMe Qualified Name (NQN). Base 2.4 caps an NQN at 223 bytes.

A host can connect to a discovery subsystem using the “well-known Discovery Service NQN”, nqn.2014-08.org.nvmexpress.discovery, the form TP4201 uses. Base 2.4 also defines a centralized discovery controller that reports what other discovery controllers and hosts register with it.

One namespace can sit behind more than one controller, and those paths may not perform alike. Asymmetric Namespace Access (ANA) reports a state for each path, such as Optimized or Non-Optimized, and multipathing software uses those states to pick a path. Ratified change TP8028 Rapid Path Failure Recovery lets a host that can reach the subsystem through another channel find out sooner that commands on a failed path are no longer being processed.

Two security pieces work in band. Hosts and controllers can authenticate each other with KX-HMAC-CHAP (called DH-HMAC-CHAP before Base 2.4). For encryption, TLS for NVMe/TCP “is based on pre-shared key (PSK) authentication” in the TCP binding text of TP8018. The NVMe over TCP Transport Specification requires an implementation that supports TLS 1.3 to support the TLS_AES_128_GCM_SHA256 cipher suite. TP8018 also strongly discourages TLS 1.2, and TP4201 extends both in-band authentication and NVMe/TCP’s TLS to post-quantum key exchange.

Hosts can also boot an operating system from a drive across the fabric. The NVM Express Boot Specification, Revision 1.4 defines extensions “that enable booting over NVMe over supported fabric transports.”

Latency the network adds

NVM Express set a goal of “no more than 10 microseconds (µs) of additional latency over a native NVMe device inside a server.” Each measurement below comes from a different setup, so they can’t be lined up as one answer. None reads a current drive locally and then over a fabric.

Sources: SPDK NVMe-oF RDMA and TCP performance reports, release 24.05, tables 21 and 22, with Guz et al., SYSTOR 2017 slides, backup slide 17, plus Klimovic et al., ASPLOS 2017, and Hwang et al., NSDI 2020, figures 6(a) and 6(b).

Both SPDK runs, the RDMA report and the TCP report, used a null block device that returns data without touching any media. Mellanox ConnectX-5 100GbE cards carried 4KiB random reads at a queue depth of 1. With no SSD in the path, those numbers isolate the software stack and the network. RDMA testing is dated June 2024 and TCP testing July 2024. On that test, TCP took about 3.7 times as long as RDMA with SPDK (17.50 / 4.72) and about 1.8 times as long with the kernel (21.39 / 12.10).

Samsung Semiconductor researchers measured 11.7µs over RoCEv2, using ConnectX-4 100GbE cards and Samsung PM1725 drives. The figure sits on a backup slide of their SYSTOR 2017 talk on the paper, and the slide labels it close to the 10µs spec target, though it is above that goal. Cornell’s i10 paper compares two remote setups with real SSDs and has no local baseline in that passage. With a real SSD in the path, drive access time becomes the main bottleneck, and the paper puts i10’s tail latency within 1.7 times that of NVMe-RDMA.

For scale, Micron’s 9650 product brief lists a typical read latency of 60µs. Against that, the null-device round trips work out to between about 8% (4.72 / 60) and 36% (21.39 / 60) of the drive’s own read time. Because it divides numbers from two different sources, this is arithmetic and no substitute for testing a 9650 over a fabric. The sources here leave gaps: none measures what Fibre Channel adds, none compares iWARP and RoCEv2 on the same rig, and none compares NVMe/TCP with local drives in production.

Checkpoints wait on the storage path

Training jobs save their state at intervals so a failure doesn’t throw away hours of work, and a job stalls while it saves. The CheckFreq paper from FAST 2021 ties the right checkpoint frequency to “model size, storage bandwidth, and training iteration time.”

The Gemini paper from SOSP 2023 puts a number on it. As an example in its introduction, the paper says checkpointing the model states of MT-NLG to remote persistent storage takes 42 minutes when that storage has 20Gbps of bandwidth. In the authors’ own testbed, the instances were connected “through a 400Gbps elastic fabric adaptor (EFA) network,” and the remote persistent storage had an aggregated bandwidth of 20Gbps. By that arithmetic (400 / 20), a single instance’s network link was about 20 times the bandwidth of the whole storage pool.

Feeding the GPUs

Data loading can starve accelerators too. Meta studied its recommendation-model training in a paper published at ISCA 2022, read here in its arXiv version. In one experiment, Meta ran a training job for its RM1 model on a single training node with eight NVIDIA V100 GPUs, two 28-core CPU sockets and two 100Gbps NICs for data ingestion, and the CPUs on that same machine prepared the data. In that run, 56% of GPU cycles were spent stalled waiting for training data. RM1 needed 16.50GB/s of data per 8-GPU training node.

For the RM2 model, moving that preparation onto separate worker machines shifted the limit to their network cards, and those workers needed about 10Gbps of their 12.5Gbps. Meta’s ZionEX training platform, a different machine from the V100 node above, gives each GPU its own 200Gbps RoCE NIC for traffic between nodes.

Keeping the KV cache across the network

Memory and Storage for SI, Part 5 covers where the key-value (KV) cache is kept once it leaves the GPU. Fetching it back from there is a network job.

Mooncake is the serving platform behind Moonshot AI’s Kimi chatbot, described at FAST 2025. It builds its cache pool on an RDMA network of up to 8×400Gbps. Its authors work out a minimum transfer rate of 6GB/s for LLaMA3-70B on a machine with eight A800 GPUs and an 8,192-token prefix. On a machine with eight H800 GPUs, that rises to 19GB/s. The paper says “a fully utilized 100 Gbps NIC per NVIDIA A800 HGX network is sufficient to meet these criteria.” After simulating bandwidths from 24Gbps to 400Gbps in section 5.4.2, the authors “recommend a minimum network bandwidth of 100 Gbps to ensure optimal system performance.”

A direct path from GPU to network card

NVIDIA’s GPUDirect RDMA “enables a direct path for data exchange between the GPU and a third-party peer device,” and network interfaces and storage adapters are among NVIDIA’s examples. Among the limits NVIDIA lists, the most important is that “the two devices must share the same upstream PCI Express root complex.” IOMMUs that translate addresses other than 1:1 must be disabled or set to pass-through. NVIDIA adds that a path with only PCIe switches between the devices performs best, one through a single CPU works with worse performance, and one across a link between two CPUs may be extremely limited or not work reliably. Where GPUs and network adapters sit on a server’s PCIe tree decides how well that direct path works.

GPUDirect RDMA covers the path between a GPU and the network card. For the path between a drive and GPU memory, NVIDIA offers GPUDirect Storage, which Memory and Storage for SI, Part 4 covers. Part 3 of this series covers the network cards that list it.

Drives per network card

NVIDIA’s ConnectX-8 SuperNIC datasheet lists a maximum total bandwidth of 800Gb/s and a PCIe Gen6 host interface of up to 48 lanes. Micron’s 9650 brief gives 28,000MB/s of sequential read for that PCIe Gen6 drive.

drives per NIC = NIC bandwidth / drive bandwidth

Here NIC bandwidth is 800Gb/s, or 100GB/s at 8 bits per byte, and drive bandwidth is 28GB/s. Dividing gives about 3.6 drives. Protocol overhead is ignored, and every drive is assumed to run flat out on sequential reads, so read it as a rough bound rather than a design rule.

When many senders hit one receiver

Storage traffic has its own congestion problem. Carnegie Mellon’s FAST 2008 incast paper found that when a client reads pieces of one data block from many servers at once, the replies can overflow the switch buffers on the client’s link, and TCP throughput collapses.

RoCEv2 relies on Priority-based Flow Control (PFC) to keep the network drop-free, as Microsoft and Mellanox put it in their DCQCN paper at SIGCOMM 2015. PFC is the amendment IEEE 802.1Qbb adds to bridged networks. The DCQCN authors found that PFC can cause head-of-line blocking and unfairness. Their fix was end-to-end congestion control built on Explicit Congestion Notification (ECN), with switch thresholds set so that ECN is always triggered before PFC.

Scaling RoCEv2 to whole data centers raised more trouble. Microsoft’s SIGCOMM 2016 paper reports PFC-induced deadlock (“yes, it happened!”), RDMA transport livelock and NIC PFC pause frame storms. One of its figures shows “the PFC pause frame storm caused by the malfunctioning NIC of one single server.” Its team answered with PFC based on DSCP priority marks, plus monitoring and management systems.

Ultra Ethernet and the work it left for later

Version 1.0 of the Ultra Ethernet Specification, dated June 11, 2025, takes another route. It “does not require or mandate changes to the network layer or the Ethernet PHY and link layers.” Its transport is built for best-effort networks that may drop packets, spraying packets across many paths with congestion control, and it can also run on lossless networks. Some optional features need support from switches, for “advanced capabilities such as packet trimming.” One lesson the spec says it takes from Ethernet and TCP/IP is that “many large-scale lossless fabrics are challenging to operate without triggering head-of-line blocking and congestion spreading.”

Storage comes up by name, and random incast events like many customers reading from one storage server “regularly occur,” in the spec’s words.

What to watch

The published record still has gaps worth watching. No source here reads the same current drive locally and then over a fabric, none measures what Fibre Channel adds, and none compares NVMe/TCP with local drives in production. The checkpoint and data-loading results above make the case for planning storage bandwidth and congestion alongside the choice of RDMA or TCP, not after it.

Ultra Ethernet is adding storage services, and the spec says that work is scheduled for a future release. Until that release, there is no Ultra Ethernet storage protocol to compare with NVMe-oF.