Technical Tutorials

High-frequency trading (HFT) operates in the realm of microseconds. At this speed, standard operating system overheads are no longer tolerable; they are fatal. To achieve consistent low latency, developers must move beyond "out-of-the-box" Linux configurations and dive deep into kernel internals. This guide explores critical optimizations for the I/O scheduler and network stack to ensure your trading application leaves no stone unturned in the pursuit of speed.

I/O Scheduler Selection and Tuning

The Linux kernel offers several I/O schedulers, each with different trade-offs between throughput and latency. For HFT, where predictable, low-latency access to storage or shared memory is critical, the Deadline or Noop (or MQ-Deadline) schedulers are typically preferred over the default CFQ or BFQ.

Why Deadline? It guarantees that I/O requests are serviced within a specified deadline, which is crucial for time-sensitive data logging or order book persistence. Noop is ideal for solid-state drives (SSDs) or NVM Express (NVMe) devices where the hardware handles its own queueing and scheduling.

# Check current I/O scheduler for /dev/sda
cat /sys/block/sda/queue/scheduler

# Set to deadline scheduler (requires root)
echo deadline > /sys/block/sda/queue/scheduler

# Increase I/O queue depth for NVMe (example)
echo 256 > /sys/block/nvme0n1/queue/nr_requests

Network Stack Optimization: Bypassing the Kernel

The traditional Linux network stack involves multiple context switches and memory copies, which are significant sources of latency. For HFT, two primary strategies are employed: TCP socket tuning and User-space networking (like DPDK or FRR).

1. Tuning the Kernel Stack

Before considering user-space solutions, optimize the kernel stack. Key parameters include disabling Nagle’s algorithm, adjusting TCP buffer sizes, and reducing queue lengths.

# Disable Nagle’s algorithm (per socket)
setsockopt(sockfd, SOL_TCP, TCP_NODELAY, &flag, sizeof flag);

# System-wide TCP tuning (example sysctl)
# Increase send/receive buffers
net.core.wmem_max = 16777216
net.core.rmem_max = 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216

# Reduce TCP reordering buffer size
net.ipv4.tcp_adv_win_scale = 1

2. Interrupt Affinity and NUMA Awareness

Network Interface Cards (NICs) generate interrupts that are handled by specific CPU cores. In multi-socket or NUMA systems, if the interrupt is handled on a core far from the application’s memory, latency spikes occur.

Bind NIC interrupts to local CPUs:

# Identify IRQ number for eth0
grep eth0 /proc/interrupts

# Bind IRQ 20 to CPU 0 (adjust for your topology)
echo 1 > /proc/irq/20/smp_affinity

Practical Example: A Low-Latency Socket Setup

Here is a C snippet demonstrating a socket configured for minimal overhead:


#include <sys/socket.h>
#include <netinet/tcp.h>

int create_optimized_socket() {
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd == -1) {
        perror("socket");
        return -1;
    }

    int flag = 1;
    // Disable Nagle
    setsockopt(sockfd, IPPROTO_TCP, TCP_NODELAY, &flag, sizeof flag);

    // Set receive buffer size
    int rcvbuf = 4 * 1024 * 1024; // 4MB
    setsockopt(sockfd, SOL_SOCKET, SO_RCVBUF, &rcvbuf, sizeof rcvbuf);

    // Enable keepalive for idle detection
    setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &flag, sizeof flag);

    return sockfd;
}

Conclusion

Optimizing for HFT is not about finding a single magic setting. It is a holistic approach involving I/O scheduler selection, kernel parameter tuning, and careful management of CPU affinity and NUMA topology. Always measure the impact of each change using precise tools like perf and custom latency probes. Remember: in HFT, every nanosecond counts, and your system configuration should reflect that obsession with precision.