As Large Language Models (LLMs) transition from experimental prototypes to production workloads, the bottleneck often shifts from GPU compute to network I/O. In high-throughput serving scenarios, a standard Linux kernel configuration is rarely sufficient. This post explores the critical infrastructure layers—specifically TCP tuning and network topology—that determine whether your model serves requests in milliseconds or seconds.
The TCP Stack Bottleneck in AI Serving
LLM inference is characterized by long-running connections for streaming responses and small, frequent requests for batch processing. The default TCP stack is designed for general web traffic, not the nuanced demands of AI inference. Key parameters often need adjustment to prevent the "Slow Start" phase from impacting latency and to maximize throughput on high-bandwidth, high-latency links.
First, we must optimize the receive and send buffer sizes. Large buffers allow the network interface to absorb bursts in traffic, preventing packet loss and subsequent retransmissions. For a typical data center link, these values should be significantly larger than defaults.
# /etc/sysctl.conf
# Enable automatic tuning for high-throughput networks
net.ipv4.tcp_window_scaling = 1
# Increase buffer sizes to 16MB (standard is often 128KB-1MB)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 87380 16777216Additionally, enabling tcp_fastopen can reduce connection setup latency for repeat requests, which is crucial for API-based services where clients maintain persistent connections.
Network Topologies: Understanding the Physical Layer
Tuning software cannot fix physical limitations. In LLM clusters, especially those using distributed inference or vLLM, the network topology between the client and the GPU server matters immensely. Avoid routing inference traffic through spine-leaf switches if it introduces unnecessary hops. Instead, aim for a Clos topology that minimizes the number of switches traversed.
For multi-node setups, consider using RDMA (Remote Direct Memory Access) over Converged Ethernet (RoCE) or InfiniBand. RDMA bypasses the kernel network stack entirely, allowing the GPU to read data directly from the network card. This reduces CPU overhead and latency by up to 50% compared to standard TCP/IP stacks.
# Example: Configuring RoCE on a Linux NIC
# Ensure PFC (Priority Flow Control) is enabled to prevent packet loss
sudo ethtool -K eth0 roce on
sudo ip link set dev eth0 type ipvlan mode l2
# Verify RoCE capability
sudo ethtool --dump-regset eth0 | grep roceApplication-Level Optimizations
Beyond the kernel and hardware, application-level tuning plays a significant role. When using frameworks like vLLM or TGI (Text Generation Inference), enable continuous batching. This allows the server to pack multiple requests into a single forward pass, maximizing GPU utilization while keeping latency low for individual requests.
Furthermore, ensure that your HTTP load balancer is configured for long-lived connections. If you are using Nginx or HAProxy, increase the proxy_read_timeout to accommodate the variable generation times of LLMs. A premature timeout is a common source of partial response errors in production LLM services.
Conclusion
Optimizing LLM inference is not just about buying faster GPUs; it requires a holistic view of the entire request path. By adjusting TCP kernel parameters, leveraging RDMA for low-latency intra-node communication, and configuring load balancers for long-duration sessions, you can significantly reduce the time-to-first-token and overall latency. Start with the network layer, measure your baseline, and iterate. The difference between a laggy user experience and a seamless one is often hidden in these configuration files.