# Translation Quality Metrics That Mean Something | Vitra.ai

> BLEU and COMET measure similarity to a reference, not fitness to publish. Four numbers that track whether quality is improving, and what each one hides.

**Canonical URL**: https://www.vitra.ai/general/translation-quality-metrics
**Source**: This is the Markdown rendering of https://www.vitra.ai/general/translation-quality-metrics, generated at build time from that page.

---

4 min read

# Translation Quality Metrics That Mean Something

BLEU and COMET measure similarity to a reference, not fitness to publish. Four numbers that track whether quality is improving, and what each one hides.

[Samhitha J Bhatt](https://www.vitra.ai/author/samhitha)
Senior Product Manager , Vitra.ai
Updated Aug 17, 2026

![Translation Quality Metrics That Mean Something](https://www.vitra.ai/static/images/blog/translation-quality-metrics.jpg)

Table of contents

[What the familiar scores actually say](#what-the-familiar-scores-actually-say)

[Four numbers worth reporting](#four-numbers-worth-reporting)

[Severity, not counts](#severity-not-counts)

[Set a baseline before changing anything](#set-a-baseline-before-changing-anything)

[FAQ](#faq)

Contributors

[Samhitha J Bhatt](https://www.vitra.ai/author/samhitha)
Senior Product Manager

Subscribe to our newsletter

Subscribe

> **Quick answer —** Automatic scores like BLEU and COMET compare output to a reference translation, which is not the same as being fit to publish. Track defect escape rate, reviewer edit distance, rework rate and time to publish instead.[Vitra.ai Universe](https://www.vitra.ai/platform) keeps one memory and one quality gate across every format.

## What the familiar scores actually say

BLEU counts n-gram overlap with a reference translation. COMET does something more sophisticated with embeddings, but the shape of the claim is the same: this output resembles a translation somebody accepted earlier. Resemblance is a useful research signal and a poor operational one. It cannot tell you whether the exclusion clause is correct, and it penalises a better translation that happens to differ from the reference.

Use them to compare two engines on the same corpus. Do not use them to decide whether to publish.

## Four numbers worth reporting

Metric

Question it answers

Watch for

Defect escape rate

What reached customers

Under-reporting when nobody logs

Reviewer edit distance

How much reviewers change

Should fall over time

Rework rate

What came back after publish

Lags by weeks

Request to published

What the business feels

Improves while quality drops

Edit distance is the most useful of the four, because it is a direct measure of whether the system is learning. If reviewers are changing as much this quarter as last, corrections are not writing back to [memory](https://www.vitra.ai/features/translation-memory) and the programme is running in place.

The last row carries a warning. Speed improves easily by reviewing less, so it should never be read alone.

## Severity, not counts

Ten typos and one inverted negation are not the same finding, and any metric that adds them together is lying.

Weight by consequence. A structured error typology — accuracy, terminology, fluency, locale convention, each with a severity — gives a defect count that means something. That is the argument for [MQM-style categories](https://www.vitra.ai/general/linguistic-qa-checklist) over a single number.

The [cultural rule engine](https://www.vitra.ai/features/quality-control) already scores this way, returning severity-weighted findings per region rather than a flat total.

## Set a baseline before changing anything

The most common reporting failure is having no before.

Measure a normal month first. Then change one thing — a threshold, a model, a [review](https://www.vitra.ai/solutions/translation-quality-and-review) band — and measure again. Changing three things at once produces a number that moved for reasons nobody can name, which is how programmes acquire superstitions.

Pair the numbers with a [judge](https://www.vitra.ai/general/llm-as-judge-translation) for triage, and keep the human review band set by risk rather than by whatever the score says. Risk is at its highest where a clause carries an obligation — [legal contract translation](https://www.vitra.ai/legal/legal-contract-translation).

## FAQ

**Are BLEU and COMET useful for translation quality?** For comparing engines on the same corpus, yes. For deciding whether to publish, no. They measure resemblance to a reference translation and penalise a better rendering that differs from it.

**Which translation quality metric matters most operationally?** Reviewer edit distance, because it shows whether the system is learning. If reviewers change as much this quarter as last, corrections are not writing back to memory and nothing is compounding.

**Why weight defects by severity?** Because ten typos and one inverted negation are not equivalent. A flat defect count that adds them together hides the only finding that mattered, so categories with severity weighting are needed.

**What is the most common mistake in quality reporting?** Having no baseline. Measure a normal month before changing anything, then change one variable at a time — otherwise a number moves for reasons nobody can identify and the team invents an explanation.

Our blog

## Lastest blog posts

Tool and strategies modern teams need to help their companies grow.

Automotive

[Automotive Brochure Localization by Market](https://www.vitra.ai/automotive/automotive-brochure-localization)
A car brochure is a spec grid, a legal footer and a photo library, all market-specific. What actually has to change, and why the layout decides the schedule.

[Samhitha J Bhatt](https://www.vitra.ai/author/samhitha)
Aug 18, 2026

Automotive

[Automotive Campaign Localization Across Markets](https://www.vitra.ai/automotive/automotive-campaign-localization)
Campaigns run through national companies and dealer networks, so one master becomes hundreds of files. Where the offer text and the disclaimers actually break.

[Samhitha J Bhatt](https://www.vitra.ai/author/samhitha)
Aug 18, 2026

Automotive

[Car Service Manual Translation for Technicians](https://www.vitra.ai/automotive/automotive-service-manual-translation)
A workshop manual is read mid-repair by someone with the car on a lift. What that demands of procedures, torque figures and fault codes, in every language.

[Samhitha J Bhatt](https://www.vitra.ai/author/samhitha)
Aug 18, 2026

[View all posts](https://www.vitra.ai/blog/page/1)

---

## Structured data

```json
{
  "@context": "https://schema.org",
  "@type": "Article",
  "mainEntityOfPage": {
    "@type": "WebPage",
    "@id": "https://www.vitra.ai/general/translation-quality-metrics"
  },
  "headline": "Translation Quality Metrics That Mean Something",
  "image": [
    {
      "@type": "ImageObject",
      "url": "https://www.vitra.ai/static/images/blog/translation-quality-metrics.jpg"
    }
  ],
  "datePublished": "2026-08-17T00:00:00.000Z",
  "dateModified": "2026-08-17T00:00:00.000Z",
  "author": [
    {
      "@type": "Person",
      "name": "Samhitha J Bhatt"
    }
  ],
  "publisher": {
    "@type": "Organization",
    "name": "Vitra.ai",
    "logo": {
      "@type": "ImageObject",
      "url": "https://www.vitra.ai/static/images/vitra-v-logo.png"
    }
  },
  "description": "BLEU and COMET measure similarity to a reference, not fitness to publish. Four numbers that track whether quality is improving, and what each one hides."
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    {
      "@type": "ListItem",
      "position": 1,
      "name": "Home",
      "item": "https://www.vitra.ai"
    },
    {
      "@type": "ListItem",
      "position": 2,
      "name": "General",
      "item": "https://www.vitra.ai/general"
    },
    {
      "@type": "ListItem",
      "position": 3,
      "name": "Translation Quality Metrics That Mean Something",
      "item": "https://www.vitra.ai/general/translation-quality-metrics"
    }
  ]
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "Are BLEU and COMET useful for translation quality?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "For comparing engines on the same corpus, yes. For deciding whether to publish, no. They measure resemblance to a reference translation and penalise a better rendering that differs from it."
      }
    },
    {
      "@type": "Question",
      "name": "Which translation quality metric matters most operationally?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Reviewer edit distance, because it shows whether the system is learning. If reviewers change as much this quarter as last, corrections are not writing back to memory and nothing is compounding."
      }
    },
    {
      "@type": "Question",
      "name": "Why weight defects by severity?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Because ten typos and one inverted negation are not equivalent. A flat defect count that adds them together hides the only finding that mattered, so categories with severity weighting are needed."
      }
    },
    {
      "@type": "Question",
      "name": "What is the most common mistake in quality reporting?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Having no baseline. Measure a normal month before changing anything, then change one variable at a time — otherwise a number moves for reasons nobody can identify and the team invents an explanation."
      }
    }
  ]
}
```