AI APIs

Building Latency-Critical Voice Agents with OpenAI's Realtime API

Oct 14, 2026 · 2 views
javascriptwebsocketsbackend developmentLLMOpenAIVoice AIRealtime API

The rise of large language models has opened the door to highly interactive AI experiences, but traditional text-based interfaces often fail to meet the immediacy expected in conversational contexts. For developers aiming to create natural, human-like voice interactions, the key lies in minimizing latency and handling bidirectional audio streams efficiently. OpenAI's Realtime API, combined with WebSockets, provides a robust foundation for building such systems. In this post, we will explore the architecture and implementation details required to deploy a low-latency voice agent.

Understanding the Realtime Architecture

Unlike standard API calls that request and receive static text responses, the Realtime API utilizes a persistent WebSocket connection. This allows for the continuous streaming of audio chunks, text deltas, and control messages between the client and server. The core benefit is the elimination of round-trip delays associated with batch processing. Instead of waiting for a complete transcript before generating a response, the model begins processing audio as it is received, enabling interruptions and real-time reactions.

The Role of WebSockets

WebSockets provide a full-duplex communication channel over a single TCP connection. In the context of voice agents, this means the client can stream microphone input while simultaneously receiving synthesized audio output. This bidirectionality is critical for handling "barge-ins," where a user interrupts the AI, requiring the system to stop generating audio and begin listening immediately.

Implementation Strategy

Implementing a voice agent typically involves three layers: the client (capturing audio), the backend proxy (managing the WebSocket connection and API keys), and the model (processing and responding). Security best practices dictate that API keys should never be exposed to the client. Therefore, a backend service is required to act as a bridge.

Backend WebSocket Proxy

Below is a conceptual example using Node.js and the ws library to manage the connection to OpenAI's Realtime endpoint. This proxy forwards audio frames from the client to the API and streams responses back.

const WebSocket = require('ws');
const { WebSocketServer } = require('ws');

const wss = new WebSocketServer({ port: 8080 });

wss.on('connection', (wsClient) => {
    // Create a new connection to OpenAI Realtime API
    const wsOpenAI = new WebSocket('wss://api.openai.com/v1/realtime', {
        headers: {
            'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`
        }
    });

    // Forward client messages to OpenAI
    wsClient.on('message', (data) => {
        if (wsOpenAI.readyState === WebSocket.OPEN) {
            wsOpenAI.send(data);
        }
    });

    // Forward OpenAI responses to client
    wsOpenAI.on('message', (data) => {
        wsClient.send(data);
    });

    // Handle disconnections
    wsClient.on('close', () => wsOpenAI.close());
    wsOpenAI.on('close', () => wsClient.close());
});

Client-Side Audio Handling

On the frontend, utilizing the Web Audio API is essential for capturing raw PCM audio at the correct sample rate (typically 24 kHz for OpenAI's models). The audio must be converted to Base64 strings and sent as JSON messages via the WebSocket. Simultaneously, incoming audio deltas must be decoded and played back using AudioContext to ensure smooth playback without buffering artifacts.

Optimizing for Low Latency

To achieve a natural conversation flow, target an end-to-end latency of under 500ms. This requires careful tuning of several parameters:

  • Chunk Size: Send audio in small, frequent chunks (e.g., 100-200ms) rather than large buffers.
  • VAD (Voice Activity Detection): Utilize the API's built-in VAD to automatically detect when the user stops speaking, allowing the model to start generating a response without waiting for a silence timeout defined in client code.
  • Connection Management: Keep the WebSocket connection alive with periodic heartbeats to prevent timeouts in intermediate proxies or CDNs.

Handling Interruptions (Barge-Ins)

One of the most challenging aspects of voice interfaces is handling user interruptions. When a user starts speaking while the AI is still talking, the AI must stop its current output and switch to listening mode. The Realtime API handles this logic server-side, but the client must actively cancel any pending audio playback when a new input_audio_buffer.speech_started event is received. This ensures that the user does not have to wait for the AI to finish a sentence before being heard.

Conclusion

Building real-time voice agents with OpenAI's Realtime API and WebSockets unlocks a new level of interactivity for AI applications. By leveraging persistent connections, streaming audio, and careful latency optimization, developers can create experiences that feel responsive and natural. As these technologies mature, we can expect even more sophisticated features like multi-party conversation handling and deeper integration with real-world tools. Start by prototyping a simple echo server, then gradually introduce the LLM layer to master the nuances of real-time audio streaming.

Share:
Related Posts