---
title: "How to test a live translation tool before you commit"
description: "Run every candidate against one fixed recording, compute word error rate, and read the error mix to locate the real problem. Plus latency and cost basis."
canonical: https://obstrans.net/en/blog/evaluating-live-translation-tools
language: en
topic: tool-reviews
published: 2026-08-08
keywords: word error rate, WER, tool evaluation, benchmark audio, live translation
---

# How to test a live translation tool before you commit

> A trial means making every candidate process the same real audio from your own studio. Word error rate gives comparable numbers, but the split between insertions, deletions and substitutions is more useful: it points at capture, noise or glossary coverage.

The least useful way to choose a live translation tool is watching demo videos. The second least useful is comparing published accuracy figures. **There is only one useful method: make every candidate process the same audio from your own studio.**

The protocol below takes about half a day. It produces comparable numbers, and something more valuable — it tells you which stage the problem is actually in.

## Why the benchmark has to be your own recording

Demo audio is clean speech at a conversational pace with no music. Live selling audio fails all three conditions: music is playing, delivery runs fast, and mouth-to-microphone distance moves around. Under those conditions the ranking between tools frequently differs from the ranking in the demos.

The requirements are specific:

- **Ten minutes of real broadcast audio**, covering explanation, pricing and interaction pacing;
- **Every brand name and model number you care about**, since this is the only exam a glossary ever sits;
- **No read-aloud recordings** — scripted reading is slower and far less slurred than live delivery.

## Capture spec: turn noise reduction off, and that is not a typo

Google's speech recognition best practices list several counterintuitive but unambiguous requirements. Record the benchmark to this spec once:

- **Sampling rate 16 kHz or higher**, and avoid resampling — if the source is natively 8 kHz, send 8 kHz rather than upsampling;
- **Use a lossless codec**, FLAC or LINEAR16 preferred; when bandwidth forces a lossy codec, AMR_WB or OGG_OPUS in that order;
- **Do not use automatic gain control (AGC)**;
- **Disable all noise reduction processing**;
- **Avoid clipping**, and position the microphone as close to the speaker as possible.

The noise reduction line is the one people read twice. Google's stated reason is that the service is built to handle noisy audio, and **applying noise reduction before sending typically reduces recognition accuracy**. Numbers measured with noise reduction enabled describe neither the tools' real capability nor your live behaviour.

## Word error rate: definition, formula, tooling

Word error rate (WER) is the industry standard. Microsoft's definition: count the insertions, deletions and substitutions, then divide by the total number of words in the human-labelled transcript.

```
WER = (I + D + S) / N × 100
```

- **Insertion (I)** — words added that were not spoken;
- **Deletion (D)** — words present in the reference but missing from the output;
- **Substitution (S)** — words replaced by different ones.

To reproduce the calculation locally, Microsoft points at `sclite` from the NIST Scoring Toolkit (SCTK). For Chinese the usual variant is character error rate, which swaps words for characters and keeps the formula and the three error categories unchanged.

**Where the bar sits.** Microsoft's general rubric: 5% to 10% is good quality and ready to use, 20% is acceptable but worth further work, and 30% or above signals poor quality. That rubric is written for general scenarios. Live selling is unusually sensitive to errors in brand names and numbers, so a practical acceptance bar normally sits tighter — how much tighter depends on your category.

## The error mix tells you more than the score

This is the part worth remembering. Microsoft documents the mapping from error type to root cause directly:

| Dominant error type | What it indicates | Where to intervene |
| --- | --- | --- |
| Mostly deletions | Weak audio signal strength | Move capture closer, check gain |
| Mostly insertions | Noisy environment, possible crosstalk | Deal with background audio and other voices |
| Mostly substitutions | Insufficient domain-specific terms | Extend the glossary or train a custom model |

That table turns an evaluation into a diagnosis. Two tools can both score 12% while one is deletion-dominated and the other substitution-dominated. The first says your microphone placement is wrong and no tool change will fix it; the second says your glossary is incomplete, which you can act on.

**Locate the fault with this table before continuing to compare tools.** If all three candidates are deletion-dominated, the real work is fixing capture, and the comparison between them means nothing yet.

## Display formatting deserves its own look

Word error rate counts words. It ignores punctuation, capitalisation and number formatting — which is precisely what your viewers read in the captions.

Microsoft defines an extended metric for this, token error rate (TER), which adds punctuation, capitalisation and inverse text normalisation differences on top of the lexical comparison. You do not need to implement it, but you do need to internalise the implication: **a tool with an excellent WER can still be unusable because it formats prices strangely.** During acceptance, pull the prices, times and sizes out of the transcript and read them. This one is visible to the naked eye.

## Latency, endurance and reconnection need separate tests

Ten minutes of benchmark audio screens out the clearly worse options. These three things it cannot touch.

**Latency.** Standardise on mouth-to-finished-translated-audio rather than mouth-to-caption. Record two local tracks, measure from the start of the original to the end of the translated playback, repeat ten times, and take the median rather than the mean.

**Endurance.** Run one session at your real broadcast length. What you are watching for is not accuracy but cumulative degradation: memory growth, drift between audio and picture, recognition getting progressively slower.

**Reconnection.** Pull the network cable once mid-session and record three things: how long the reconnect takes, whether sentences are lost, and whether the host sees any indication at all. The last one gets overlooked most often — if the tool fails silently, a host can spend ten minutes talking to an audience receiving no translation whatsoever.

## Record the cost basis, not the price

Prices change, and a price written into an evaluation sheet is wrong within months. Record the basis instead:

- Billing per minute, per character, or by subscription?
- Is speech synthesis billed separately? Is a cloned voice extra?
- How is overage charged?
- Is there a concurrency limit?

Normalise all of it to direct cost per broadcast hour on the same basis, and only then compare.

## A results table you can copy directly

Record these columns per candidate, and the decision makes itself:

| Column | What goes in it |
| --- | --- |
| Word / character error rate | Overall score on the shared benchmark |
| I / D / S split | Count for each error category |
| Brand name hits | How many target terms came out correct |
| Price formatting | Whether any rendering is unacceptable |
| Median latency | Mouth to finished translated audio, median of ten |
| Endurance | Any degradation across one full-length session |
| Reconnection | Recovery time, sentences lost, host-visible warning |
| Cost basis | Billing unit and what is excluded |

Once the table is filled in, you no longer hold a vague impression of which tool is better. You hold a set of facts you can weigh against your constraints — and whichever column fails tells you whether the next move is fixing hardware, extending the glossary, or changing tools.

## Steps

1. **Record a representative benchmark clip** — Capture ten minutes from a real broadcast covering explanation, pricing and interaction pacing, and containing every brand name and model number you care about. Do not use a read-aloud script; its pace and elision are nothing like live delivery.
2. **Record it to the documented capture spec** — Google's speech recognition best practices call for a sampling rate of 16 kHz or higher, no resampling, a lossless codec such as FLAC or LINEAR16, no automatic gain control, and all noise reduction disabled. Record once to that spec and reuse the file for every candidate.
3. **Transcribe it by hand** — Write out an accurate reference transcript sentence by sentence. This text is the denominator for every number that follows, so a mistake in it contaminates the whole comparison. It is worth proofreading twice.
4. **Run every candidate against the same file** — Process the identical clip with each tool and export both the recognised text and the translation. Conditions have to match exactly — same file, same glossary configuration, same language direction. One difference invalidates the comparison.
5. **Compute word error rate and split the error types** — Word error rate is the sum of insertions, deletions and substitutions divided by the total words in the reference. To reproduce the calculation locally, Microsoft points at sclite from the NIST Scoring Toolkit. A single score is not enough; you need the count for each of the three categories.
6. **Read the error mix to locate the stage at fault** — Microsoft documents the mapping directly. Many deletions usually mean weak audio signal strength, so move capture closer. Many insertions mean a noisy recording environment with possible crosstalk. Many substitutions mean insufficient domain-specific terms.
7. **Test latency, endurance and reconnection separately** — Measure mouth-to-finished-translated-audio with a two-track local recording, repeated ten times, taking the median. Then run one full-length session and pull the network cable once, noting how long recovery takes and whether sentences are lost.
8. **Record the cost basis rather than the price** — Write down whether billing is per minute, per character or per subscription, whether speech synthesis is included, and how overage is charged. Prices change and the basis does not, and only the basis makes candidates comparable.

## FAQ

### What word error rate is good enough?

Microsoft's general rubric is that 5% to 10% is good quality and ready to use, 20% is acceptable but worth improving, and 30% or more signals poor quality. Live selling is more sensitive to errors in brand names and numbers, so a practical acceptance bar usually sits tighter than the general rubric.

### Why does the vendor demo look so much better than my own test?

Demo audio is typically clean speech at conversational pace with no background music, and live audio fails all three conditions. That is exactly why the benchmark has to be your own recording — the only number worth anything comes from your own studio.

### Should noise reduction be on when recording the benchmark?

No. Google's best practices explicitly say to disable all noise reduction processing and automatic gain control, noting that pre-processing typically reduces recognition accuracy because the service is designed to handle noisy audio. A number measured with noise reduction on does not predict live behaviour.

### Is a ten-minute sample enough?

It is enough to screen candidates against each other, not to conclude anything absolute. Ten minutes reliably separates the clearly worse options, but deciding whether one tool will hold up long term also needs the endurance and reconnection tests, which a short sample cannot surface at all.