As Large Language Models (LLMs) scale to hundreds of billions of parameters, the bottleneck shifts from compute to communication. In the world of AI infrastructure, network latency and bandwidth are no longer just metrics; they are the determining factors in time-to-train and time-to-inference. For engineers optimizing GPU clusters, choosing the right protocol stack is as critical as selecting the right transformer architecture. This post dissects the three pillars of high-performance networking—TCP, UDP, and RDMA—and evaluates their suitability for modern AI workloads.
Understanding the Baseline: TCP and UDP
For decades, the Internet Protocol Suite has relied on Transmission Control Protocol (TCP) and User Datagram Protocol (UDP). While ubiquitous, their default configurations often struggle with the massive data throughput required by distributed training.
TCP (Transmission Control Protocol) guarantees delivery and order. This reliability comes at a cost: the "head-of-line blocking" effect and significant CPU overhead due to kernel context switches. In LLM training, where gradient synchronization happens millions of times per second, TCP's reliability features can introduce jitter that stalls GPU pipelines.
UDP (User Datagram Protocol) is connectionless and lightweight. It offers low latency but lacks error correction. Some AI frameworks use UDP for control planes or specific all-reduce operations because it reduces overhead, but it requires complex application-layer logic to handle packet loss, which is non-trivial to implement correctly at scale.
The Game Changer: RDMA over Converged Ethernet (RoCE)
RDMA (Remote Direct Memory Access) revolutionizes data center networking by allowing a network interface card (NIC) to exchange data directly between the memory of one machine and another, bypassing the operating system kernel entirely. This technique, known as Kernel Bypass, drastically reduces CPU utilization and latency.
In the context of LLMs, RoCE (RDMA over Converged Ethernet) enables:
- Microsecond Latency: Critical for inference where every millisecond counts for user experience.
- Zero-Copy Data Transfer: Eliminates data copying between kernel and user space, preserving CPU cycles for model execution.
- High Throughput: Efficiently moves the terabytes of gradient data needed during distributed training.
Practical Configuration for RDMA
Implementing RDMA requires specific hardware (InfiniBand or RoCE-enabled NICs) and careful configuration. Below is an example of checking RDMA device status on a Linux node, a essential first step in debugging AI networking issues.
# List available RDMA devices
ibstatus
# Check if RoCE is enabled on the specific interface
ethtool -k eth0 | grep rdma
# Sample output might look like:
# RDMA VLAN offload: onWhen configuring your cluster, ensure that your ibdev2netdev mappings are correct and that Quality of Service (QoS) settings prioritize RDMA traffic to prevent packet loss in congested networks.
Choosing the Right Protocol: Training vs. Inference
The choice between protocols depends heavily on the workload phase:
Training Workloads
Distributed training involves massive collective communications (All-Reduce, All-Gather). Here, bandwidth is king. RDMA/RoCE is the superior choice for large-scale training clusters (e.g., training a 70B+ parameter model). It minimizes the "communication barrier," ensuring GPUs spend more time computing and less time waiting for data. TCP can be used for smaller models or single-node fine-tuning where setup complexity outweighs the benefits.
Inference Workloads
Inference is often request-driven and highly variable. While RDMA provides the lowest latency, the overhead of setting up RDMA connections for every inference request can be prohibitive unless using connection pooling or specific hardware offloading. For many inference scenarios, TCP with optimized kernel settings (such as increasing net.core.somaxconn) and hardware offloading (TCP segmentation offload) provides sufficient performance with significantly lower operational complexity.
Conclusion
As AI models grow, so does the complexity of the underlying network stack. While TCP remains a reliable default for general computing, and UDP offers speed with complexity, RDMA stands out as the necessary evolution for high-performance AI infrastructure. For organizations scaling to hundreds of GPUs, investing in RDMA-capable hardware and optimizing the network stack is not just an option—it is a prerequisite for efficient model development and deployment.