Translation Quality Metrics That Mean Something

Quick answer — Automatic scores like BLEU and COMET compare output to a reference translation, which is not the same as being fit to publish. Track defect escape rate, reviewer edit distance, rework rate and time to publish instead.

Vitra.ai Universe keeps one memory and one quality gate across every format.

What the familiar scores actually say

BLEU counts n-gram overlap with a reference translation. COMET does something more sophisticated with embeddings, but the shape of the claim is the same: this output resembles a translation somebody accepted earlier. Resemblance is a useful research signal and a poor operational one. It cannot tell you whether the exclusion clause is correct, and it penalises a better translation that happens to differ from the reference.

Use them to compare two engines on the same corpus. Do not use them to decide whether to publish.

Four numbers worth reporting

Edit distance is the most useful of the four, because it is a direct measure of whether the system is learning. If reviewers are changing as much this quarter as last, corrections are not writing back to memory and the programme is running in place.

The last row carries a warning. Speed improves easily by reviewing less, so it should never be read alone.

Severity, not counts

Ten typos and one inverted negation are not the same finding, and any metric that adds them together is lying.

Weight by consequence. A structured error typology — accuracy, terminology, fluency, locale convention, each with a severity — gives a defect count that means something. That is the argument for MQM-style categories over a single number.

The cultural rule engine already scores this way, returning severity-weighted findings per region rather than a flat total.

Set a baseline before changing anything

The most common reporting failure is having no before.

Measure a normal month first. Then change one thing — a threshold, a model, a review band — and measure again. Changing three things at once produces a number that moved for reasons nobody can name, which is how programmes acquire superstitions.

Pair the numbers with a judge for triage, and keep the human review band set by risk rather than by whatever the score says. Risk is at its highest where a clause carries an obligation — legal contract translation.

FAQ

Are BLEU and COMET useful for translation quality? For comparing engines on the same corpus, yes. For deciding whether to publish, no. They measure resemblance to a reference translation and penalise a better rendering that differs from it.

Which translation quality metric matters most operationally? Reviewer edit distance, because it shows whether the system is learning. If reviewers change as much this quarter as last, corrections are not writing back to memory and nothing is compounding.

Why weight defects by severity? Because ten typos and one inverted negation are not equivalent. A flat defect count that adds them together hides the only finding that mattered, so categories with severity weighting are needed.

What is the most common mistake in quality reporting? Having no baseline. Measure a normal month before changing anything, then change one variable at a time — otherwise a number moves for reasons nobody can identify and the team invents an explanation.