Networking for SI, Part 3: DPUs and SmartNICs as Storage Engines

A data processing unit (DPU) is a network interface card (NIC) with its own processor, memory and fixed-function engines, so storage work that would run on a server’s CPU can run on the card instead. NVIDIA’s BlueField-3 carries up to 16 Arm cores beside up to 400Gb/s of networking, and NVIDIA’s DOCA SNAP software uses that card to show the host an NVMe drive that is really a volume across the network.

Part 1 of this series covered how NVMe over Fabrics (NVMe-oF) carries NVMe commands between servers. In this part the card speaks it instead of the host, whether as the initiator behind an emulated drive or as the target inside a storage box, and it can encrypt and check data on the way through. Every measurement cited here comes from research prototypes on Broadcom’s Stingray SmartNIC rather than BlueField or AMD’s Pensando cards, so each result below names the hardware it ran on.

A processor on the network card

Storage software takes real host time. Experiments by the researchers behind LeapIO suggest that 10–20% of data center x86 cores may have to be reserved for storage functions. Those cores would otherwise run customer work, so a card that runs the same software on processors of its own hands them back.

NVIDIA’s ConnectX-7 shows where an ordinary NIC stops. Its datasheet lists NVMe-oF storage target offloads and AES-XTS data-at-rest encryption as hardware functions, but no Arm cores for software of its own. A DPU adds them. Research papers call the same kind of card a SmartNIC, and this part uses both words for a network card that runs storage code on its own processors.

BlueField-3’s datasheet lays out one such card. Network ports sit on one side, and up to 32 lanes of PCIe Gen 5.0 face the host on the other, through a PCIe switch that can bifurcate into up to 16 downstream ports. The datasheet also lists M.2 and U.2 connector options for drives attached to the card itself. Between the two sides sit the Arm cores, 16GB of DDR5, a programmable datapath accelerator of 16 cores and 256 threads, and engines for encryption and erasure coding.

AMD’s Pensando line splits the work differently. Its second-generation Elba DPU builds its data path from 144 P4-programmable match processing units (MPUs) at 2GHz, beside 16 Arm A72 cores at 3GHz. With the third-generation Salina, AMD moves to PCIe Gen5 and 400GE ports. Their makers’ documents give enough to set the three cards side by side.

Source: NVIDIA BlueField-3 DPU datasheet; AMD Pensando Elba and Salina product briefs. Salina’s brief does not give its core or data path engine counts, so those cells are blank.

Both makers name storage among these cards’ jobs. BlueField-3 carries BlueField SNAP for NVMe and VirtIO-blk plus acceleration for NVMe-oF and NVMe/TCP, and Salina offers NVMe virtualization with NVMe-oF over RDMA or TCP. Core counts and port speeds say little about how well a card does that job, and the only measurements below come from an older Broadcom card.

NVMe emulation: a local drive that lives across the network

NVIDIA’s DOCA SNAP-4 Service Guide describes the idea plainly. SNAP presents networked storage as a local block device, “emulating a local drive on the PCIe bus.” A host operating system or hypervisor loads its standard NVMe driver and never learns that no physical drive sits behind the function.

Each emulated function is tied to a storage controller in the SNAP service on the card. That controller answers most admin commands itself and passes I/O commands to a backend device, built here on SPDK (the Storage Performance Development Kit). Backends can be NVMe-oF, iSCSI or other protocols, over remote direct memory access (RDMA) or TCP, on Ethernet or InfiniBand. SNAP can also move data by zero-copy direct memory access (DMA) from the remote storage straight to host memory. So the fabric choice from Part 1 moves off the host and onto the card, while the host still sees an ordinary NVMe drive.

NVIDIA runs that data path on small cores of its own. By default SNAP uses the data path accelerator (DPA) rather than the Arm cores, and DPA is a cluster of RISC-V processors inside BlueField. With PCIe switch emulation turned on, BlueField can present up to 31 hot-plugged NVMe or virtio-blk functions. SNAP on BlueField-2 needed a license per card, while BlueField-3 needs none, according to the guide. Salina can be deployed the same way, with DPUs on compute servers emulating NVMe virtual functions alongside network and security services.

Researchers built and measured the idea before these products reached the market. LeapIO was presented at ASPLOS 2020 and exposes virtual NVMe storage to unmodified guest virtual machines (VMs). Its authors built a development board around the Broadcom StingRay V1 system on chip (SoC), which pairs a 100Gb Ethernet NIC with eight Arm Cortex-A72 cores at 3GHz, and also ran the runtime in a SoC-like VM on x86. In that VM, the runtime cost 2–5% of throughput against bare metal and still delivered 0.65 million I/O operations per second (IOPS)IOPS counts commands finished per second. Queue depth is how many commands are outstanding at once, and a drive needs a deeper queue to reach its top IOPS. on a data center SSD.

On the real SoC, the same runtime ran up to 30% slower for a local SSD than in that VM. Its authors trace that to an RDMA crossing of 5µs on every access to the host’s queues, and to Arm cores clocked 25% lower than the x86 cores. Those numbers belong to a 2020 prototype on Stingray, and none of the sources here measures SNAP on BlueField-3 itself.

They still show where emulation spends its time. Each hop between the card and host memory costs microseconds. With the real SoC on the remote side only, throughput fell just 5% because the remote runtime never had to cross back to its own host, and the authors expect overheads like the local case once both sides run on the SoC.

Running the NVMe-oF target on the card

At the far end of the fabric, the card can be the target instead. In a SmartNIC JBOF (just a bunch of flash), a carrier board holds the SmartNIC, a PCIe switch and the drives, and the NVMe-oF target runs on the SmartNIC’s cores instead of an x86 server’s. University of Washington and Samsung researchers studied that design for Gimbal at SIGCOMM 2021, on a Broadcom Stingray PS1100R. That card has eight Arm A72 cores at 3.0GHz, 8GB of DDR4-2400 and a 100Gb Ethernet NIC.

Measured at the wall, the Stingray storage node draws 52.5W at most. That is a little over a quarter of the 192.0W a Xeon-based storage server drew (52.5 / 192.0). It delivered 1.4 million 4KB random read IOPS at 75.3µs of unloaded latency. On both platforms the slowest phase was NVMe command execution, which runs from the submission-queue write through the SSD’s own work to the completion, so the two nodes ran close.

With one I/O outstanding, the SmartNIC added 1.0% to random read latency on average for requests up to 64KB, and 20.3% and 23.3% at 128KB and 256KB. Throughput took more cores rather than more time. A Xeon server reached 1,513 KIOPS (thousand IOPS) of 4KB random reads with two cores, and the SmartNIC served similar read and write traffic with three Arm cores. Gimbal’s own design sits on top of that hardware. It schedules many tenants on one JBOF, and its prototype reached up to 6.6 times better utilization and 62.6% less tail latency.

NVIDIA also sells the target as a fixed function. ConnectX-7 offers NVMe-oF storage target offloads among its storage accelerations, so a storage server can pass part of the target’s work to its NIC without moving the whole target onto card cores.

How much time a card can spend on each I/O

Gimbal’s authors also measured how little time those Arm cores have to spare. Serving about 3,000MB/s of 4KB sequential writes, the 99.9th percentile latencyA latency percentile gives the time within which that share of commands finished. The 99th percentile is the time that 99% of commands beat. was 34µs on the server and 66µs on the SmartNIC. With all of the card’s cores busy, the SmartNIC could add at most 1µs of processing to each 4KB read and 5µs to each 4KB write before bandwidth fell. At 128KB per request, the margin grew to 5µs for reads and 10µs for writes.

At the same bandwidth, a stream of 128KB requests carries 32 times fewer requests than a 4KB stream (128KB / 4KB), so each request can absorb more work. Against that budget, the 5µs RDMA crossing LeapIO paid on its Stingray SoC equals the whole allowance for a 4KB write on Gimbal’s card, though the two papers ran different workloads and neither measured that comparison.

This guide reads that as the case for the fixed engines every card here lists. Software on a SmartNIC’s cores can steer I/O, schedule tenants and run an NVMe-oF target. Work that touches every byte of a small block would likely fit in a microsecond only when an engine built for it does the work.

Encryption and checksums on the way through

Inside a self-encrypting driveA self-encrypting drive encrypts all data with AES in hardware as it is written, using a media key that the drive itself holds., the controller encrypts data with a key that never leaves it, as Flash Foundations Part 5 explains. That protects the flash, but on a fabric the data travels to the drive unencrypted unless something earlier in the path encrypts it.

SNAP’s bundled SPDK supports hardware encryption offload. Its crypto block device encrypts write commands and decrypts read commands on the card with AES-XTS, so the volume behind it, and the fabric between, carry only ciphertext. BlueField-3 and ConnectX-7 both carry AES-XTS 256/512-bit data-at-rest encryption in hardware.

Zero copy comes with a condition. SNAP keeps its zero-copy flow with encryption only for an NVMe backend over RDMA. With another transport or backend, the guide says extra DMA operations first move the data from the host to the BlueField Arm side. AMD’s Salina brief names data-at-rest encryption among its storage features as well.

Integrity work goes to engines too. Salina offers acceleration for CRC64 and CRC32 cyclic redundancy checks (CRCs)A cyclic redundancy check is a check value sent with data so the receiver can detect errors. On a PCIe link, a failed check makes the sender replay the data., plus deduplication hashes. Elba’s offload engines handle checksums, RAID and deduplication hashing, while BlueField-3 names erasure coding for RAID and data integrity among its storage uses.

NVMe Protection Information (PI) is the standard way to carry a checksum with every block from host to flash. Flash Foundations Part 8 notes that it protects the path only when every adapter and fabric in it takes part. None of the datasheets and briefs cited here names NVMe PI, so whether a card’s checksum engines carry PI end to end is a question for each vendor’s documentation.

What to watch

A measurement is still missing. LeapIO and Gimbal tested Stingray cards with eight Arm A72 cores. BlueField-3 has up to 16 Arm A78 cores plus its RISC-V data path accelerator, and nothing cited here reports how SNAP’s emulated drive compares with a local one in latency or IOPS. A published test with one I/O outstanding, run on the same SSD locally and behind SNAP, would show whether newer silicon closes the microsecond gap Gimbal measured.

Three threads tie into the rest of the site. BlueField-3 also lists NVIDIA GPUDirect StorageGPUDirect Storage is NVIDIA’s path that moves data straight between an NVMe drive and GPU memory, skipping a copy through the CPU’s memory., the drive-to-GPU path that Memory and Storage for SI Part 4 covers, beside the GPU-to-NIC path from Part 1. NVIDIA’s BlueField-4 powers the context memory storage platform that Part 5 of that series follows. Salina advertises “UEC Ready RDMA,” which points at the Ultra Ethernet transport that the next part of the series weighs against RoCE.

Judged on the published numbers, the storage box is where a SmartNIC has the clearest case today. With one I/O outstanding, its random reads up to 64KB took 1.0% longer than a server’s, on a node that drew a little over a quarter of the power. The emulated drive on the host side has no independent test yet.