Networking for SI, Part 5: CXL and SDXI Where They Touch Storage

A Compute Express Link (CXL)CXL runs over the same physical layer as PCIe but adds cache-coherent memory access, so it moves cache lines rather than blocks of storage. memory module has no flash translation layerThe flash translation layer is the firmware that maps host addresses to physical flash locations and hides the rules of erase blocks and wear. and never appears as an NVMe namespace, yet it sits on the storage path of an SI cluster in three places. Checkpoints can land there before they reach SSDs, KV cacheA KV cache stores the key and value vectors a language model computes for each token, so later tokens reuse them instead of recomputing them. It grows with context length. blocks can cross between servers through it without a network card in the hop, and data bound for flash gets copied through it. SNIA’s Smart Data Accelerator Interface (SDXI) is the standard written for those copies.

The Working Memory guide to CXL memory explains what CXL memory is, and Flash Foundations Part 10 tracks which versions are deployed. Part 1 of this series followed NVMe commands across the network. This part stays on the memory side of the same traffic and covers where CXL and SDXI meet storage: staging, the cache’s trip between servers, and the copies between tiers.

Memory on the far side of a link

CXL carries load and store instructions over the same PCIe wires an SSD uses. A read from CXL memory is a cache-line access the processor issues as it would to a dual in-line memory module (DIMM), while a read from an SSD is a block command that passes through a queue, the drive’s firmware and its flash.

Microsoft’s Pond study estimates that CXL adds 70 to 90ns over DRAM on the same non-uniform memory access (NUMA) node for pools of 8 to 16 sockets, and more than 180ns at rack scale. Real devices vary by controller. Sun and colleagues measured three CXL memory devices whose load latency ran from 35% to 3x longer than DDR5 on the remote socket. Those figures sit on the latency chart of the memory and storage hierarchy guide.

In the TraCT study covered below, a Niagara 2.0 Type 3 expander measured 640ns and 10.1GB/s with Intel’s Memory Latency Checker. Micron’s 9650 product brief lists a typical read latency of 60µs for that PCIe Gen6 SSD. The two figures come from different setups, so they set a rough scale between the tiers and no more.

CXL memory sits between host DRAM and the local SSDs on the path data drains along, on the load and store side of the line. DRAM on CXL keeps nothing at power-off, so a checkpoint or cache block parked there still has to reach flash.

Flash placed behind a DRAM front tier on CXL is a different design, and Memory and Storage for SI, Part 2 covers it. Whether a given block belongs in memory or on flash is a cost per access question, and Memory and Storage for SI, Part 7 redoes the five-minute rule for current parts. The host DRAM tier above CXL has module changes of its own, which Working Memory, Part 6 covers.

Expansion, pooling and sharing

CXL can attach that memory in three ways, and each one changes who can reach it. Expansion is one host and one module. CXL 2.0 adds switching and memory pooling, as the consortium’s release for that specification describes, so a module’s capacity can be handed out to more than one host. Its 3.0 white paper separates the two ideas, since pooling allocates and frees memory between hosts while sharing lets more than one host reach the same region at once, kept coherent in hardware.

Version 3.0 also reaches past one switch. It supports up to 4,096 nodes with port based routing, and a Global Fabric Attached Memory (GFAM) device can be reached by up to 4,095 of them.

Hardware runs behind that text. TraCT’s authors report that when several hosts attach to one Type 3 device, coherence covers at most small device-specific regions, and current devices offer no atomic operations across nodes, so their software builds its own locking. Alibaba Cloud’s Beluga reached a pool through a real CXL 2.0 switch. Its switch node holds two XConn XC50256 chips, each forwarding 2TB/s over 256 PCIe 5.0 lanes, and connects up to 16 servers to an 8TB pool with 1TB/s of total bandwidth. Flash Foundations Part 10 keeps the deployment status of pooling silicon and CXL 3.x platforms.

For storage planning, a shared CXL pool today reads as pooled memory with software coordination on top, because that is what both papers had to build. TraCT adds its own locking, and Beluga’s authors keep shared KV cache blocks consistent on their CXL 2.0 switch by flushing a writing host’s caches out to CXL memory.

Checkpoints land in memory first

A checkpoint is a training job’s state written out, and a SNIA webinar on storage for SI training, given on October 30, 2024 by presenters from Celestica and Dell, sizes it from the parameter count alone. At 2 bytes per parameter for weights in the BF16 number format and 12 more for optimizer and other state, GPT-3’s 175B parameters make a 2.4TB checkpoint and MT-NLG’s 530B make 7.4TB.

Training pauses while a checkpoint is saved, and the same webinar shows where the pause goes. With PyTorch’s torch.save, GPUs sit idle through the copy from GPU memory to host memory and then through the flush to disk. Gemini, Meta and torchsnapshot use in-memory checkpointing instead, which holds the GPUs only for the copy into host memory and flushes to storage while training runs.

SSDs still receive the whole checkpoint, but the flush leaves the critical path. As Part 1 showed, in the Gemini paper’s example saving MT-NLG’s model states to remote storage with 20Gbps of aggregate bandwidth takes 42 minutes. Gemini sizes those model states its own way, so its time and the webinar’s checkpoint size come from different accounting and don’t divide into each other.

Staging only helps when the copy fits in the tier. Host DRAM is the tier these schemes use, and none of the papers read for this part puts the staging copy on CXL memory instead. A CXL pool adds capacity to that tier on the memory side of the PCIe link, which makes checkpoint staging the most direct storage use of CXL in this guide’s reading, and also the one with no published measurement among these sources.

The KV cache crosses the fabric

Memory and Storage for SI, Part 5 covers the tiers a KV cache moves through, from GPU memory to remote storage. This section covers only the hop between servers, and today that hop runs over remote direct memory access (RDMA), as Part 1 showed for Mooncake.

That traffic is small and scattered. A talk on KV cache offloading at SNIA’s 2025 Storage Developer Conference describes fixed-size blocks from 100KB to 100MB and up, written once and never changed, with offloading libraries tuned for MB-scale I/O. Inside a block it gets finer. In Qwen-32B, one 16-token block takes 128 non-contiguous 20KB transfers, according to Beluga’s authors.

TraCT, from Virginia Tech and SK hynix America, replaces that hop with CXL shared memory under NVIDIA’s Dynamo. GPUs write KV blocks into the shared region and read them back with loads, stores and direct memory access (DMA), so the network card leaves the path. On two servers, each with one NVIDIA A6000 GPU and 512GB of DRAM, it cut the average time to first token (TTFT) by up to 9.8x and the 99th-percentile (P99) latency by up to 6.2x. Peak throughput rose by up to 1.6x against an RDMA path over a 100Gbps network card and a DRAM cache. Beluga reports an 89.6% cut in TTFT and 7.35x the throughput in vLLM against RDMA-based designs, and up to 4.79x the throughput of Mooncake.

Both are research systems measured against baselines their own authors chose, and TraCT’s rig is two servers. Both remove the same step. TraCT’s authors describe an RDMA transfer moving each block through network card queues and host DRAM buffers on both ends, and Beluga’s describe a bounce buffer in host memory. That puts CXL’s first change to storage design on the cache’s trip between servers rather than on its size.

SDXI, one interface for memory copies

Moving data between tiers is mostly copying, and a CPU core running a copy loop does much of it today. SNIA’s SDXI specification is “a standard for a memory-to-memory data mover.” The interface is written to be independent of whose data mover sits behind it, so existing drivers keep working on later hardware without changes. Its revision history dates version 1.0 to November 28, 2022. SNIA announced it on February 16, 2023, written by a work group of 89 people from 23 member companies, and version 1.0a followed on October 3, 2024.

Its queue will look familiar from Part 1. Each SDXI descriptor is a 64-byte entry in a circular ring. Software places descriptors, moves a write index forward and rings a doorbellA doorbell is a register on an NVMe drive. Writing the new tail of a queue there tells the drive a command is waiting., and the data mover reads them and moves a read index. Once a connection is set up, a user-mode process can copy from one address space to another without privileged software in the loop. Every piece of state sits in registers or memory, so a hypervisor can stop a data mover and move it during live migration.

Storage is on the roadmap more than in the 1.0 text. At SNIA’s 2024 developer conference, the work group’s chair listed persisting data and tiered memory among SDXI’s sample uses. The chair named storage use cases with NVMe and computational storage, data integrity checks with cyclic redundancy checks (CRC) and Protection Information, and CXL use cases as v1.1 candidates, in a list marked as a draft. As of November 2026, the work group page, last updated August 2025, shows an update to version 1.0 open for public review and v1.1 still to come.

Hardware and software support are arriving. StorageReview reports that AMD’s 6th Gen EPYC (Venice) adds SDXI to offload memory copies. An AMD engineer posted a Linux driver for PCIe-hosted SDXI 1.0 devices in September 2025, as a request for comments.

What to watch

SDXI v1.1 is the release to watch for storage. If its NVMe use cases and data-integrity checks reach the final text, a copy engine could move a checkpoint from a CXL tier toward an SSD and check it on the way, under one standard driver. SDXI 1.0 covers memory-to-memory copies only.

For CXL, the gap is between the specification and the devices. CXL 3.0 describes hardware-coherent sharing across hosts, while TraCT and Beluga built their coordination in software on today’s hardware. A published measurement of a checkpoint staged on CXL memory, and a KV cache test on more than two servers, would show whether the gains those papers report hold at cluster scale.

Where Memory Goes Next sorts memories by the role each one fills, and on the storage path CXL is a tier that some traffic can pass through on its way to flash. The Networking for SI series page lists every part.