Can an LLM Judge Translation Quality?

Quick answer — An LLM can judge translation quality well enough to triage what a human should see, provided it grades against a rubric with the source in front of it. It is unreliable as a final verdict, and it favours fluency over accuracy.

Vitra.ai Universe keeps one memory and one quality gate across every format.

What it is good for

Triage. Not certification. Given a source, a translation and a specific rubric, a model identifies the segments most likely to contain a fault with enough reliability to order a review queue. That is genuinely useful, because ordering the queue is most of the value of reviewing at all.

What it cannot do is tell you the translation is correct. Absence of a flag is not evidence.

Three failure modes worth knowing

Fluency bias is the serious one, because it is the same blind spot a monolingual human reviewer has — which means stacking an LLM judge on top of a target-language review does not add an independent check. Both miss the same thing.

That is the argument for pairing it with back translation, which fails differently.

Structuring a judge that behaves

Four things make the difference between a usable signal and noise.

Give it the source. A judge grading the target alone is grading fluency, which is the failure mode you are trying to avoid. Give it a fixed error typology rather than a ten-point scale, so it names a category instead of inventing a number. Ask for the offending span, because a judgement that cannot point at anything is usually not a judgement. And use a different model from the one that produced the translation.

Then calibrate against human ratings on a sample you already trust, and re-check when the model changes underneath you.

Where it fits

As one input among several, wired into the run rather than bolted on afterwards by a person.

Agentic workflows can route on its output, and the proofreading and back-translation agents catch different things, and a cultural rule check catches something else again — a verdict built from several disagreeing signals is stronger than any single score.

Anything with contractual or clinical consequence still goes to a human. An LLM judge changes which items reach that person, not whether the person is needed. Which documents are read and which are relied on is set out in machine translation in legal work.

FAQ

Can an LLM reliably score translation quality? It is reliable enough to order a review queue and unreliable as a final verdict. Given the source and a fixed rubric it identifies likely faults well, but absence of a flag is not evidence of correctness.

What is fluency bias in LLM evaluation? The tendency to score smooth but inaccurate output above accurate but awkward output. It is the same blind spot a monolingual reviewer has, so the two checks are not independent of each other.

How should an LLM judge be set up? Give it the source text, a fixed error typology rather than a numeric scale, and a requirement to quote the offending span. Use a different model from the one that produced the translation.

Does an LLM judge replace human review? No. It changes which items reach a human rather than whether one is needed. Contractual, clinical and regulated content still requires a person regardless of what any automated judge reports.