The latency budget of live translation: where the seconds go

“How much latency is there” is the first question everyone asks and the one most often answered vaguely, because the number depends entirely on where you start and stop measuring.

Agree on the definition first

The same system produces answers several seconds apart under two common definitions:

• Speech to caption — from the host’s first syllable to the translated caption appearing on screen.
• Speech to end of spoken translation — from the host’s first syllable to the synthesised voice finishing that sentence.

The second is the moment an overseas viewer has actually understood, and it is the operationally meaningful number. Everything below uses it.

Five stages

[1] segmentation wait → [2] recognition → [3] translation → [4] synthesis → [5] playback

Stage 1: segmentation wait. The system has to decide whether the sentence is finished. The usual mechanism is voice activity detection: N milliseconds of continuous silence marks a boundary.

This is the one stage with published numbers to anchor against. Azure AI Speech calls the parameter the segmentation silence timeout, documents its range as 100 to 5000 milliseconds with 500 as a typical default, and states the trade-off directly: higher values produce longer results and tolerate longer pauses mid-phrase but make results take longer to arrive, while lower values break phrases up more promptly and can split a single sentence across several results. The two worked examples in that documentation happen to be exactly the two ends of live selling — try 300 ms when fast delivery chains sentences together, try 2000 ms when the speaker pauses inside a sentence.

This stage is a product decision, not a technical limit, and it sets the floor for everything else: those 500 milliseconds start counting after the host has finished the sentence.

Stage 2: recognition. Streaming recognition produces results while the host is still speaking, so its incremental cost is often small — it runs in parallel with speech. The real expense is waiting for a stable final result, because streaming recognisers continuously revise their earlier guesses and translation needs something settled.

Stage 3: translation. The fastest of the five. How many milliseconds depends on the vendor and where you sit on the network, and there is no figure worth quoting generically — but its share of the total is small enough that it does not deserve to be optimised first. You can verify that yourself: time a single sentence sent straight to the translation API, then compare it against the end-to-end figure.

Stage 4: synthesis. Generating natural speech takes time, and cloned voices take more. Cost here scales roughly with the length of the translated text.

Stage 5: playback. Consistently forgotten, and it is pure physics: a sentence of a dozen or so English words takes several seconds to say at a normal pace, regardless of how fast your stack is. This cannot be optimised away, only shortened by shortening sentences. To find your own figure, read a typical translated segment aloud with a stopwatch — more reliable than any estimate.

What this implies about priorities

Once you see the distribution, the optimisation order is obvious:

1. Shorten sentences. This affects stages 1, 4 and 5 simultaneously, so it has the largest payoff. It is a scripting change, and it is the single most effective thing a host can do.
2. Tune segmentation. Find the balance between shredding and waiting. There is a better lever than the number itself: since Speech SDK 1.41, Azure offers semantic segmentation (set Speech_SegmentationStrategy to Semantic), which the documentation describes as segmenting mainly on sentence-ending punctuation, specifically to fix the two failure modes of pure silence detection — long unbroken speech that never gets split, and short pauses that split in the wrong place. The limits are on the same page: not all languages and locales are covered, and confidence scores and NBest lists are not yet supported.
3. Adjust synthesis rate. A small increase directly shortens stage 5, and it is a standard parameter rather than a hack: Google Cloud Text-to-Speech’s speakingRate accepts 0.25 to 2.0, where 1.0 is the voice’s native speed. Start at 1.1 and listen before committing — how much naturalness you lose depends on the voice and the language, and there is no general answer.
4. Only then consider a different translation engine.

Most teams do this in reverse: two weeks benchmarking translation APIs, followed by the discovery that latency did not move.

One cost that hides at the end

Beyond those five stages there is one more: delivery buffering. OBS and the streaming platform each buffer, so what the audience sees is already several seconds behind the host.

The useful property is that this delay applies equally to original and translated audio. It does not change how far the translation trails the original voice, only the absolute wall-clock offset. Factor it into interaction design, such as answering comments; ignore it when chasing audio sync.

Measuring it yourself

No special equipment required:

1. Start a local recording with the original and translated tracks on separate channels;
2. Say something with a sharp plosive — a “three, two, one” countdown works well — so it is easy to locate on the waveform;
3. Open the recording in any audio editor and measure from the start of the original to the end of the translation.

Repeat ten times and take the median, not the mean. Occasional long tails make the average meaningless.