OpenAI has launched GPT‑Realtime, an end-to-end speech model without intermediate conversion to text. According to OpenAI, GPT‑Realtime was developed with customer service, personal assistants, and education in mind.

 

OpenAI has previously provided speech applications (including Whisper, TTS, Voice mode in ChatGPT), but these were solutions that required intermediate conversion to text.

GPT‑Realtime was announced in May 2024 and is now available in production. GPT‑Realtime processes speech in a single pass, and according to OpenAI, this would latency should be significantly reduced. However, OpenAI does not disclose the achievable latency. It does claim that the model can handle subtle elements of speech well, such as intonation, pauses, emotion, and even non-verbal cues like laughter.

Demo

This allows GPT‑Realtime to deliver more natural-sounding speech with nuance in intonation, pace, and emotion. The model can also follow instructions such as "speak quickly and professionally" or "speak with an empathetic French tone." OpenAI is also introducing two new voices, Marin and Cedar, and updating eight existing voices with more expressive features. The OpenAI news release also includes a demo that shows the possibilities.

According to OpenAI GPT Realtime scores better It performs against its predecessors on various benchmarks, both in terms of comprehension and intelligence, as well as conversation handling and contextual understanding. The model can also handle images, allowing you to engage in dialogue about what the model sees. Finally, the model supports SIP telephony, enabling phone-based voice agents.

Latency

According to a technical report from the platform latent.space The so-called time-to-first-byte via the Realtime API (from OpenbAI) averages around 150 milliseconds for customers in the United States; on top of that, there's the latency of GPT-Realtime, which would need to be significantly lower than 500 milliseconds for a viable solution. The key question is whether GPT-Realtime can deliver on that.

Speech-to-speech solutions need to take into account processing time factors such as end pointing (determining when the application can start working – this requires a certain amount of silent time to decide when to switch turns), buffering (necessary for a stable audio stream), network delay, processing the audio, and generating output.

What latency is acceptable?

In traditional VoIP systems, latency of 20–150 milliseconds is generally considered acceptable; up to 200 milliseconds is considered natural by people. Between 200 and 500 milliseconds, people experience some delay but it is considered acceptable. Higher values ​​are perceived as slow, although values ​​up to 1000 milliseconds can be workable in specific applications. Values ​​above that (1 second or more) are considered annoying. (OpenAI, Ziptone)

Technology