Goal
Inside one GPU server the GPUs talk over NVLink. Between servers they talk over a network, and for training jobs that network has to move large amounts of data with low delay and almost no loss. Two technologies are used for it: InfiniBand, a network built for this from the start, and RoCE, which carries the same kind of memory-to-memory transfers over Ethernet. Both depend on RDMA, remote direct memory access, where a network card reads or writes memory on another machine without the CPUs copying the data.
I do not have this equipment, so this post is reading and arithmetic. The sources are NVIDIA’s reference architecture and product pages and a general description of RoCE, all linked below. The point is to know the vocabulary of these fabrics, how they are laid out, and what to check when one misbehaves.
Terms used in this post
- RDMA: transferring data between the memory of two machines through the network cards, with little CPU involvement.
- InfiniBand: a switched network standard with its own link, transport and management layers, used in many GPU clusters.
- RoCE: RDMA over Converged Ethernet, which carries RDMA over Ethernet. RoCE v2 runs over UDP and IP, so it can be routed.
- Lossless network: one tuned so it does not drop packets under load. InfiniBand does this with credit-based link flow control, and Ethernet fabrics usually use priority flow control.
- Congestion control: how senders slow down when the network is filling. RoCE v2 uses ECN marks and congestion notification packets.
- Adaptive routing: letting switches pick a less busy path for a flow instead of one fixed path.
- Fat tree: a switch layout with leaf switches next to the servers and spine switches above them, designed so bandwidth does not shrink toward the top.
- Rail: the network path used by one network card position across all servers. In an eight-card server, there are eight rails.
- Rail-aligned: a design where the card in position n of every server connects to the leaf switch for rail n.
- Subnet manager: the InfiniBand service that discovers the fabric and programs paths into the switches.
- GPUDirect RDMA: letting the network card read and write GPU memory directly, without a copy through host memory.
What I built
- A diagram of a rail-aligned fabric. Servers with one network card per GPU, leaf switches by rail and spine switches above.
- A short calculator.
bench/part7-arithmetic.py fabricworks out per-node and per-GPU bandwidth, and the size of a SuperPOD scalable unit, from figures in NVIDIA’s reference architecture.
How it works
With RDMA, an application hands the network card a description of a transfer, and the card moves the data between the registered memory of two machines. The CPU is not in the data path. NVIDIA’s GPUDirect RDMA documentation describes the same idea with GPU memory as one end, so the data goes from a GPU to a network card without being staged in host memory first. This is why a GPU and its network card sitting under the same PCIe switch matters, which Part 8’s first post showed in the topology matrix.
InfiniBand and RoCE differ below the RDMA layer. Wikipedia’s RoCE article summarizes the differences: InfiniBand uses a credit-based mechanism to guarantee lossless transfers between adapters, while RoCE runs on Ethernet and may need the Ethernet to be configured as lossless, typically with priority flow control. InfiniBand’s congestion control is based on FECN and BECN marking, and RoCE v2’s uses ECN marks and congestion notification packets. RoCE v2 uses UDP destination port 4791. The same article notes that configuring a lossless Ethernet network can be more complex than configuring InfiniBand. NVIDIA’s Quantum InfiniBand page lists adaptive routing, congestion control, and in-network aggregation (SHARP) among its features and says it supports topologies such as fat tree. NVIDIA’s Spectrum-X page describes its Ethernet platform as RoCE with extensions, aimed at lossless networking, in two-tier leaf and spine layouts. The Ultra Ethernet Consortium is an industry effort to specify Ethernet transport for AI and HPC workloads.
NCCL ties the pieces together on the host. Its environment variable reference includes NCCL_IB_HCA, which chooses which RDMA interfaces NCCL uses, and a set of RoCE settings, so one library runs over either fabric.
NVIDIA’s DGX SuperPOD reference architecture for DGX H100 shows how it comes together. It uses four separate networks: a compute fabric, a storage fabric, an in-band management network and an out-of-band management network. The compute fabric is InfiniBand and rail-aligned in groups of 32 servers. Traffic on one rail is one hop from the other 31 servers in its group, and traffic between rails or between groups goes through the spine layer. The storage fabric is also InfiniBand, and the document states the per-server storage bandwidth must exceed 40 GB/s.
Commands and results
From the vendor figures the calculator reports the following. A DGX H100 has eight network cards of up to 400 Gbps each, listed in the user guide, which is 3,200 Gbps or 400 GB/s toward the compute fabric, and 50 GB/s per GPU. Compared with the 900 GB/s of NVLink per GPU, the network gives each GPU one eighteenth of that. A scalable unit is 32 servers and 256 GPUs. Four units would be 128 servers, and the reference design for the full system has 127 servers, 1,016 GPUs, because the document says one system is removed to make room for the fabric manager’s connectivity. The storage requirement of over 40 GB/s per server is 320 Gbps, 0.8 of one 400 Gbps card, so storage is a separate network from the compute fabric and not a share of it.

None of this was run. It is arithmetic over documented figures.
Challenges
- A lossless network that is almost lossless is worse than a lossy one. RoCE needs priority flow control and ECN set consistently on every switch and card. One switch with a different setting causes pauses or drops that show up as slow collectives, with no error in the job.
- One slow link slows the whole job. In a collective, every GPU waits for the slowest transfer. A single degraded cable or a card that negotiated a lower speed makes a whole group of servers slow, so link speed and error counters are part of every health check.
- Rails make wiring mistakes expensive. In a rail-aligned design, a cable in the wrong leaf port sends that card’s traffic through the spine, where it competes with everything else. The fabric works, and performance quietly drops.
- A subnet manager is a service to run. InfiniBand needs one to discover the fabric and program routes, so it needs monitoring, failover and an upgrade plan like any control-plane service.
- The lab has no way to practice this. There are no RDMA cards or switches here, and software emulation of RDMA cannot show congestion behavior. In the future I’m planning to purchase a hardware kit to help practice further.
Differences with a production setup
- Scale and design. The numbers above are for one reference design, described in the SuperPOD reference architecture. Other sites choose other oversubscription and topology.
- Ethernet alternatives. Cloud providers and some operators run RoCE on Ethernet, which the Spectrum-X page and the Ultra Ethernet Consortium describe.
- Management. Production fabrics are cabled and checked with tools from the switch vendor and monitored for link errors, congestion and cable faults. Cabling and port maps live in an inventory such as the NetBox setup from Part 6.
- Host side. NCCL settings, adapter selection and GPUDirect RDMA configuration all matter and vary with the platform.
FAQ
What does RDMA do?
It moves data between the memory of two machines through the network cards, with the CPUs out of the data path.
How do InfiniBand and RoCE differ?
InfiniBand defines its own transport, with credit-based flow control. RoCE runs RDMA over Ethernet and usually needs the Ethernet set up as lossless with priority flow control and ECN.
What is a rail?
The path of one network card position across all servers. A server with eight cards connects to eight rails.
Why are there separate compute and storage networks?
Each has its own bandwidth needs, and keeping them apart stops storage traffic from slowing gradient exchanges.
What will change the most between this reading and a real fabric?
Behavior under load. Here the fabric is described from documentation. On real switches and cards, congestion and flow control decide what a job gets, and that is what you would measure first.
Repeating with real hardware
Coming soon. With a pair of RDMA-capable servers I will run bandwidth and latency tests between them, then check how congestion and a bad link show up in an NCCL all-reduce test.
Next
Take the quiz to test your understanding: Part 8 quiz
The next post, reference architectures, covers how these pieces are packaged and sold.
Comments