Eyes on Flash/ Driven by Insight
Contact

Search Eyes on Flash

Tip: press / or Ctrl+K on any page to open search.

Type to search.

    Technical guide

    Networking for SI

    The networks between SI servers and their storage: how NVMe commands cross them, what published tests say they add in latency, and where congestion sets in.

    A series on the networks that carry storage traffic for SI clusters, in five parts. Part 1 covers NVMe over Fabrics: how NVMe commands travel as capsules over RDMA, TCP and Fibre Channel, and what published tests say the network adds in latency. Part 2 maps the four networks in an SI cluster, why storage stays off the GPU fabric, and how to size the storage network. Part 3 covers DPUs and SmartNICs, network cards with their own processors that run the storage work for the host. Part 4 sets Ultra Ethernet against RoCEv2. Part 5 covers where CXL memory and the SDXI data mover meet storage. How NVMe works inside one server is in Flash Foundations, and the memory and storage built for SI is in Memory and Storage for SI.

    The series

    Read in order

    Each part stands on its own.

    NVMe over Fabrics: a local SSD on a short PCIe link, the same SSD reached across a switch with capsules on the wire, and many senders converging on one receiverA three-panel schematic. Left: a host chip sits directly above an SSD, joined by one short copper link. Center: a host on the left and a storage box with SSDs on the right sit on either side of a network switch, and small teal capsules travel along the copper link between them. Right: four sender boxes on the left send copper lines into one switch, which feeds a single receiver box on the right.GUIDE / NETWORKING FOR SI, PART 1NVMe across the networkLOCAL · FABRIC · INCAST01 LOCAL02 FABRIC03 INCASTHOSTSSDHOSTcapsulesSSDsswitchmany to one

    Part 1

    NVMe-oF and the Network Behind SI Storage

    How NVMe over Fabrics carries NVMe commands over RDMA, TCP and Fibre Channel, what published tests say it adds in latency, and how SI storage traffic loads it.

    Read Part 1
    The four networks in an SI cluster: scale-up inside each rack, the back-end between racks, storage, and the front-endA row of six racks of GPU trays. A copper bar runs down the middle of each rack for its scale-up links. A dashed copper line above the racks joins them all for the back-end network. Below the racks, a teal line runs to a block of storage on the right, and a dashed mint line runs to a block for users and management on the left.GUIDE / NETWORKING FOR SI, PART 2Four networks, one clusterSCALE-UP · BACK-END · STORAGE · FRONT-ENDBACK-ENDSCALE-UPFRONTENDSTORAGE

    Part 2

    The Four Networks in an SI Cluster

    What the four networks in an SI cluster carry, why storage stays off the GPU fabric, and how to size it from MLPerf Storage's checkpoint test.

    Read Part 2
    DPUs and SmartNICs: a card that shows the host an NVMe drive, a SmartNIC that serves a box of SSDs, and an engine that encrypts data on its way throughA three-panel schematic. Left: a host chip sits above a DPU card; the card holds four small processor cores beside a dashed box marked NVMe, the drive the host sees, and a dashed copper link leaves the card for the fabric. Center: a storage box with a SmartNIC on top, a PCIe switch below it and four SSDs at the bottom, with the fabric link entering from the left. Right: three solid data blocks pass through an AES-XTS engine on the card and leave as three hatched blocks.GUIDE / NETWORKING FOR SI, PART 3Storage engines on the network cardEMULATE · TARGET · ENCRYPT01 EMULATE02 TARGET03 ENCRYPTHOSTNVMeto the fabricSmartNICbox of SSDsAES-XTSplaincipher

    Part 3

    DPUs and SmartNICs as Storage Engines

    How a network card with its own processor shows the host a local NVMe drive, runs the NVMe-oF target and encrypts data, and what published tests say it costs.

    Read Part 3
    RoCEv2 against Ultra Ethernet: one lossless path held by pause frames, and one connection sprayed across every path with drops allowedA two-panel schematic. Left, RoCEv2: a sender, two switches and a receiver in a row on one copper path; a queue of teal bars fills above the second switch, and dashed copper arrows labeled pause run backward under the path toward the sender. Right, Ultra Ethernet: a sender on the left fans out over four switches stacked in a column to one receiver on the right; a small teal packet rides each path and one packet on a lower path is cut short.GUIDE / NETWORKING FOR SI, PART 4No drops, or every pathROCEV2 · ULTRA ETHERNET01 ROCEV202 ULTRA ETHERNETpausepauseone path, no dropsevery path, drops allowed

    Part 4

    Ultra Ethernet Against RoCE

    What Ultra Ethernet changes against RoCEv2: its own transport, packet spraying and congestion control, and which products are sampling, in market or qualifying.

    Read Part 4
    CXL and SDXI where they touch storage: a checkpoint staged in CXL memory before flash, a KV cache crossing between servers through shared CXL memory, and a descriptor ring for memory copiesA three-panel schematic. Left: a GPU box at the top, a teal CXL memory box in the middle and a teal SSD at the bottom, joined by copper arrows pointing down. Center: two server boxes on either side of one mint shared memory region inside a teal CXL device, each server linked to it by a copper line. Right: a ring of eight slots, three filled teal, above two buffers joined by a copper copy arrow.GUIDE / NETWORKING FOR SI, PART 5CXL and SDXI on the storage pathSTAGE · CROSS · COPY01 STAGE02 CROSS03 COPYGPU

    Part 5

    CXL and SDXI Where They Touch Storage

    Where CXL memory and SNIA’s SDXI data mover meet storage in SI clusters: checkpoint staging, the KV cache’s trip between servers, and copies between tiers.

    Read Part 5

    All 5 parts

    1. Part 1NVMe-oF and the Network Behind SI StoragePublished
    2. Part 2The Four Networks in an SI ClusterPublished
    3. Part 3DPUs and SmartNICs as Storage EnginesPublished
    4. Part 4Ultra Ethernet Against RoCEPublished
    5. Part 5CXL and SDXI Where They Touch StoragePublished

    On this site

    Where it connects

    The guides link these pages where they lean on them.

    Continue on this site

    Flash Foundations All Guides Timelines