Networking for SI, Part 4: Ultra Ethernet Against RoCE

RDMA over Converged Ethernet (RoCE) carries the InfiniBand transport across Ethernet and depends on a network that does not drop packets. Version 1.0 of the Ultra Ethernet Specification is dated June 11, 2025. It replaces that transport with one that expects drops, recovers from them, and can spread each flow across every path through the network.

That difference decides how a storage network for an SI cluster is built and tuned. Part 1 showed why RoCEv2 networks lean on pause frames and what those pauses cost at scale. This part compares the two transports on loss recovery, path use, and congestion handling. It then sorts the products that cite the Ultra Ethernet specification by what each maker says about them today.

What RoCEv2 expects from the network

Remote direct memory access (RDMA) lets one machine read or write another machine’s memory without that machine’s CPU copying the data. RoCEv2 is the routable version of RoCE. On September 16, 2014, the InfiniBand Trade Association announced RoCEv2, which let RoCE traffic cross Layer 3 networks using standard IP address routing. Each packet travels inside a User Datagram Protocol (UDP) datagram. The IANA port registry lists UDP port 4791 as roce and assigns it to the InfiniBand Trade Association.

Inside that wrapper sits the transport InfiniBand clusters use. Mittal and colleagues at UC Berkeley describe it in their SIGCOMM 2018 paper Revisiting Network Support for RDMA. A receiver that gets a packet out of order discards it and sends a negative acknowledgment (NACK). Its sender then retransmits every packet sent after the last one acknowledged, and that scheme is called go-back-N. RoCE kept the design, and operators made the network lossless with Priority-based Flow Control (PFC) so those retransmissions would rarely happen.

PFC is the amendment IEEE 802.1Qbb adds to bridged networks. According to the same Berkeley paper, PFC brings head-of-the-line blocking, congestion spreading, and occasional deadlocks. Large RoCEv2 deployments also use Explicit Congestion Notification (ECN), which RFC 3168 defines for IP, so senders slow down before PFC has to pause a link. The DCQCN paper sets switch thresholds so ECN always fires first, and Part 1 covers that history.

AMD adds a second limit in a May 6, 2026 post on its networking transport. By its account, RoCEv2 keeps each connection on a single path, so equal-cost multipath (ECMP) routing cannot spread one connection’s traffic, and go-back-N resends the whole window after a single loss. A maker is describing a rival design here, though its go-back-N account matches the paper’s.

A transport built to lose packets

The Ultra Ethernet Consortium (UEC) wrote the Ultra Ethernet Transport (UET) as a replacement. In the specification’s words, UET “sets out to address the shortcomings of RoCEv2” in semantics, the transport layer, wire operations, implementation complexity, and scale limits. Applications reach it through the libfabric programming interface instead of the InfiniBand verbs RoCE uses.

UET still rides on ordinary IP. It runs over UDP and may also run directly over IP as an experimental option. A path-choosing entropy value goes in the UDP source port so existing switches can hash it. When version 1.0 was published, IANA had not yet assigned UET a UDP port or an IP protocol number, so both fields stay configurable.

Four sublayers make up the transport. A semantic sublayer turns sends, reads, and writes into packets. Reliability and ordering belong to the packet delivery sublayer. A congestion management sublayer paces traffic and picks paths, and an optional transport security sublayer encrypts and authenticates it.

Reliability comes in four delivery modes, which differ in ordering, duplicates, and congestion control:

Section 3.2.2 of the Ultra Ethernet Specification v1.0 defines these modes.

RUD carries the bulk data. Its packets may arrive in any order, and the delivery sublayer drops duplicates so each one touches memory only once. ROD keeps message headers and other ordered traffic in sequence, and RUDI suits tiny messages that can safely be written twice. Because RUDI and UUD skip UET congestion control, the specification asks that they not share a traffic class with the other two.

UET also groups features into profiles. A high-performance computing (HPC) profile has the most, and the AI Full and AI Base profiles are subsets of it for the messaging AI training clusters use. Its first target is the backend scale-out network, and its listed workloads include storage traffic next to AI and HPC work.

Spraying packets over every path

A data center fabric has many equal paths between two servers. Switches choose among them with ECMP, hashing header fields so that every packet of one flow follows one path. According to the Ultra Ethernet specification, single-path transports such as RoCE handle this badly when two large flows hash onto the same link, and it cites several research papers for that finding.

UET keeps ECMP in the switches and moves the choice to the sender. Its congestion management sublayer sets the entropy value in each packet. Switches are expected to send packets with the same entropy value down the same path, so changing the value changes the path. Under UET, a connection carrying unordered traffic may use all paths to the destination at once. Spreading packets this way is called packet spraying, and it is why RUD accepts any arrival order.

Spraying changes how loss is found. A RoCE receiver treats a gap as a loss and triggers go-back-N. Under UET, an acknowledgment can carry a 64-bit selective acknowledgment bitmap with one bit per packet sequence number, so the sender resends only what is missing. On cost, the specification is frank. Packets on a congested path can arrive much later than their neighbors, so counting out-of-order arrivals is a weak loss signal, and probing every path to find a loss costs more.

For a storage network this is where the two designs part most. Checkpoint writes and dataset reads are bulk transfers, which is the traffic RUD exists to carry unordered. The specification also expects incast. It says random events where many customers read one storage server “regularly occur,” and Part 1 covers incast from the storage side.

Congestion control without pause frames

PFC works link by link. Lossless mode, as the specification lays it out, starts when a switch’s receive buffer passes a threshold. That switch sends a PFC pause out the port the packet came in on, and the upstream switch stops sending. Loss stays near zero, but pauses can spread upstream, saturate whole trees of switches, and in the worst case deadlock.

UET moves the control to the endpoints. Network-signal based congestion control (NSCC) runs mainly at the sender and reads only round-trip time and ECN marks. Receiver-controlled congestion control (RCCC) runs at the receiver, which watches incoming flows and grants them credit. According to the specification, RCCC is expected to work best on nonblocking fat trees and pairs with NSCC where the tree is oversubscribed.

Switches help in two ways. UET asks them to mark ECN as a packet leaves the queue rather than when it enters. A switch may also trim a packet it cannot buffer. It cuts the packet short, changes its differentiated services code point (DSCP), and forwards the stub to the receiver as a congestion signal. Trimming is optional in switches, but every endpoint must accept trimmed packets.

Lossless operation stays available. On a lossless network, UET should put requests and acknowledgments in two separate traffic classes so that pausing requests cannot deadlock the replies. Link Layer Retry (LLR) handles errors on a single link instead of end to end. Credit-Based Flow Control (CBFC) is the other link-layer addition, and the specification leaves every link-layer feature optional for compliance, LLR and CBFC included.

Version 1.1 arrived on October 12, 2026 with Congestion Signaling (CSIG). Switches update this 4- or 8-byte Layer 2 tag with the congestion each packet met, and both NSCC and RCCC read it. That release also adds programmable congestion management and a verbs mapping for the AI Base profile, so applications written for verbs can be ported with fewer changes.

Sampling, in market, or being qualified

Three network cards cite the Ultra Ethernet specification by name. Each maker uses its own label and gives its own status:

Each row comes from AMD’s blog posts of June 12, 2025 and May 6, 2026, or from Broadcom’s news release of October 14, 2025.

In a June 12, 2025 post, AMD called Pollara the industry’s first UEC-ready AI network interface card (NIC) and said racks built with Pollara 400 cards were in market. The same post previewed Vulcano as the next-generation card, UEC 1.0 compliant and set to support 800G of network throughput.

AMD’s May 6, 2026 post on its networking transport says Pollara has been validated at scale in AMD labs with Instinct MI350 and MI355 clusters, co-validated with OpenAI, running a transport called Multipath Reliable Connection (MRC). The same post says Vulcano is still being qualified. It describes MRC as spraying packets, resending only lost packets, and using NSCC based on the UEC 1.0 specification.

Broadcom announced Thor Ultra as sampling on October 14, 2025, listing packet-level multipathing, out-of-order delivery, selective retransmission, and programmable congestion control. Its March 12, 2026 OFC release still calls the card UEC-compliant and gives no newer availability statement. On the P1800GO product page, Broadcom describes the same features on its PCIe 6.0 x16 card as advanced RoCE.

UEC publishes no list of tested products. Its compliance page offers a test-bed guide, profile matrices, and checklists, and the accompanying readme says they are provided for self-attestation. Every compliance label in the table is therefore the maker’s own.

Each maker’s status stays its own. By AMD’s account, racks with Pollara are in market and its MRC transport uses NSCC from the 1.0 specification. Thor Ultra, whose spraying and selective retransmission Broadcom’s product page calls advanced RoCE, is sampling, and Vulcano is still being qualified. On this guide’s reading, the specification’s mechanisms are reaching buyers piece by piece inside network cards, and no maker in these sources says it sells a full UET stack end to end, with libfabric on the hosts and switches that trim packets.

What to watch

Storage is the gap for this series. UEC’s 1.0 specification schedules its storage services for a future release. October’s announcement of version 1.1 lists scale-up support, collectives, and congestion signaling but no storage services. Until that work appears, NVMe over Fabrics keeps running over RoCE or TCP as Part 1 describes. In these sources, Ultra Ethernet’s mechanisms reach a storage network as features of network cards, with Pollara in market, Thor Ultra sampling, and Vulcano still being qualified.

Next, look for a maker statement that Thor Ultra or Vulcano has reached volume production, an IANA port for UET, and switches that trim packets in production fabrics. Photon Fabric tracks the co-packaged optics switches built for the same fabrics.