Technical Tutorials

Building Large Language Model (LLM) training clusters is less about buying individual GPUs and more about architecting the connections between them. As models scale from billions to trillions of parameters, the "network wall" becomes the primary bottleneck. Engineers must decide between NVIDIA’s proprietary NVLink for tight-coupling and Ethernet (specifically InfiniBand or RoCE) for scaling out. This post breaks down the technical trade-offs to help you design the right topology.

Understanding the Bandwidth Landscape

The fundamental difference lies in throughput and latency. NVLink provides a high-bandwidth, low-latency fabric directly between GPUs, effectively making multiple GPUs behave like a single, massive memory unit. Ethernet, while vastly improved with technologies like 400GbE or 800GbE and RDMA (Remote Direct Memory Access), introduces higher latency due to the switch layer and protocol overhead.

For Data Parallelism, where each GPU holds a copy of the model and processes different batches of data, Ethernet is often sufficient. However, for Model Parallelism or Pipeline Parallelism—where activation tensors must be exchanged frequently during backward passes—NVLink’s superior bandwidth prevents the compute units from idling while waiting for data.

Topology Considerations

When designing your cluster, you need to consider how GPUs are grouped. A common misconception is that you can simply string GPUs together over Ethernet for everything. Here is a practical breakdown:

NVLink Topology: Typically used within a single node (e.g., 8x H100 GPUs connected via NVSwitch). This creates a "super-GPU" effect.
Ethernet/InfiniBand Topology: Used for inter-node communication. This is where you connect thousands of nodes together.

For optimal performance in distributed training frameworks like PyTorch Distributed Data Parallel (DDP) or DeepSpeed, you should aim for NVLink within nodes and high-speed InfiniBand (using NCCL backend) between nodes.

Implementation Example: Configuring NCCL

When setting up your training script, the environment variable NCCL_IB_DISABLE dictates which backend is used. To ensure you are leveraging the high-speed interconnect, you must disable NVLink if you are testing pure network latency, or enable InfiniBand for inter-node comms.

# Example: Force NCCL to use InfiniBand for inter-node communication
# This is crucial for multi-node LLM training performance
export NCCL_SOCKET_IFNAME=eth0  # Fallback interface
export NCCL_IB_HCA=mlx5        # Specify the InfiniBand adapter
export NCCL_DEBUG=INFO         # Check initialization logs

# Run your training job
python -m torch.distributed.run \
    --nproc_per_node=8 \
    --nnodes=4 \
    --node_rank=$RANK \
    --master_addr=$MASTER_ADDR \
    --master_port=$MASTER_PORT \
    train.py

Always monitor your NCCL logs. If you see frequent warnings about "NCCL WARN net/socket failed," it indicates your Ethernet fallback is kicking in, which will severely degrade training throughput.

Cost and Scalability Trade-offs

NVLink is expensive. It requires specialized PCIe switches and topology within the server chassis. Ethernet, particularly using standard 400Gb/800Gb Ethernet switches, offers a more cost-effective path to scale out to hundreds or thousands of nodes. However, the "cost" of Ethernet is often paid in developer time and tuning effort. Optimizing RDMA settings, buffer sizes, and congestion control algorithms for Ethernet is significantly more complex than plugging into an NVSwitch fabric.

Conclusion

There is no one-size-fits-all answer. For small-scale fine-tuning (<100 GPUs), NVLink-only clusters are often overkill, and standard Ethernet suffices. For foundational model pre-training, NVLink is non-negotiable for intra-node communication, while high-performance Ethernet/InfiniBand is critical for inter-node scaling. By understanding these distinct roles, you can build an LLM infrastructure that balances cost, complexity, and raw computational power effectively.