Live translation: from recognition to synthesis

“Live translation” sounds like one feature. It is four independent systems chained together:

Microphone → [segmentation / VAD] → [ASR] → [machine translation] → [TTS] → output

Each stage has its own latency and its own error rate, and errors propagate forward. If recognition hears “thirty-nine” instead of “thirty-nine ninety-nine”, translation will not fix it and synthesis will read the wrong number out in a perfectly natural voice.

Where the latency actually sits

Most people assume translation dominates. In practice, on a common pair like Chinese to English, machine translation is usually the fastest of the four stages. The real cost is at both ends:

• Segmentation wait. The system has to decide whether a sentence has finished. Wait too little and you shred sentences; wait too long and you add pure latency. That trade-off is a product decision, not a technical limit — Azure’s segmentation silence timeout is adjustable from 100 to 5000 ms with a typical default of 500 ms, which tells you the order of magnitude involved.
• Speech synthesis. Generating natural speech takes time, especially with a cloned voice, and the translation is not really “delivered” until playback finishes. Long sentences stretch perceived latency far more than the numbers suggest.

So if you want lower latency, look at segmentation policy and synthesis first, not at swapping translation engines.

Where the accuracy actually goes

The accuracy bottleneck is almost always the first stage. A live audio environment is hostile to speech recognition: background music, fast sales patter, regional accents, product names and model numbers embedded in ordinary speech. Once recognition is wrong, nothing downstream can recover.

The most effective quality improvements are therefore usually not a stronger translation model, but:

• Better capture — Google’s speech recognition best practices put “microphone as close to the speaker as possible” and “disable noise reduction and automatic gain control” on the same list, and the second half is routinely done backwards;
• A glossary loaded up front with brand names, model numbers and category jargon, so both recognition and translation know them;
• A small concession on pace: live sales delivery runs faster than conversational speech, and fast delivery chains several sentences into a single recognition result.

What this topic covers

We take the four stages one at a time: how each is usually implemented, which metrics you can actually measure, and how live differs from conference interpreting. Most conference-grade practice does not transfer, because live streaming is one-way, continuous, and offers no chance to pause and repeat.