Gains:
- Ability to distinguish between automatic QA and linguistic LQA layers and apply both in the correct order
- Ability to objectively evaluate the translation according to category and weight (critical/major/minor) with an error framework such as MQM
- Being able to make the final decision when using AI as an auditor, knowing its blindness to its own translation, the risk of making mistakes, and the limits of automated metrics
The moment you say a translation is "done", that's half the work; The other half is to prove that that translation is actually reliable. In this unit, you will learn systematic ways to measure translation quality: automated QA checks, human-based LQA error frameworks, quality metrics, and how to use AI as an “inspector” — and its limits. The aim is to move away from the subjectivity of "I think it's good" and become an expert who defines, measures and proves quality.
Two types of quality control: automatic and linguistic
QA (Quality Assurance) works in two layers in translation:
Automated QA: Stylistic errors that CAT tools and scripts catch — number mismatch, missing/excess space, inconsistent term, untranslated segment, bad placeholder/tag, double space, punctuation. These are scanned quickly and completely by machine; It escapes the human eye, but the vehicle catches it.
LQA (Linguistic Quality Assurance): This is the layer where a human evaluator examines meaning, terminology, style, grammar and local appropriateness. Automated QA "does the number match?" looks at the question; LQA “is the meaning correct, is the tone appropriate, is it culturally appropriate?” He looks at the question. The two complement each other; One does not replace the other.
Tip: Always run Auto QA first; Cleaning up formal errors allows the human evaluator to devote attention to real linguistic problems. A reviewer who bothers with the number error will miss the tone error.
Error frames: MQM and DQF
Standard error typologies are used to make quality measurable rather than “good/bad.” The most common is MQM (Multidimensional Quality Metrics): it assigns each error to a category (accuracy, fluency, terminology, style, locality, format) and a weight (critical, major, minor). Another framework is DQF (Dynamic Quality Framework) and is mentioned together with MQM.
Why is it important? Because with this framework, you can give an error score to a translation, objectively compare different translators/engines and ask "can this text be delivered?" You answer the question with a numerical threshold. For example: critical error = 10 points, major = 5, minor = 1; text below a certain threshold passes per certain word.
Critical error: error that inverts meaning, creates security/legal risk, damages brand (wrong dose, “not responsible” instead of “responsible”). Major: disruptive but harmless. Minor: noticeable but not detracting from the meaning (minor style/punctuation).
Using AI as a controller — and its limits
AI is fast and helpful in checking a translation against the error framework: you can say “mark correctness, terminology, fluency errors in this translation with MQM categories”. It reduces your blind spots and gives you a quick "second eye".
But there are three critical limits: (1) AI can be blind when checking the translation it produces — both making and confirming the same mistake; cross-check with a different model or return to the source. (2) AI can also make up hallucinatory “errors” — report an error that does not exist; Confirm each warning. (3) the AI may not fully grasp the real-world severity of a critical error (a dose error can be fatal); The expert does the weighting. So AI speeds up QA, but the final quality decision and delivery approval lies with the human.
Caution: Saying "AI has passed QA, it is clean" is a false sense of confidence. AI auditing is not a replacement for human LQA and comparison with the source; it's a pre-screening layer that gets them up to speed.
three mini cases
Case 1 — Automated QA found 40 number errors. In a financial report translation, automated QA caught 40 number/format mismatches between source and target (thousand separator, decimal, currency). The human eye would miss most of these; vehicle listed in seconds.
Case 2 — MQM score led to engine selection. One bureau MQM-rated the output of two different MT engines at 2,000 words: Engine A 12 error points, Engine B 31. Objective measurement settled the "which is better" debate with a score; The bureau made Engine A the standard and planned its post-revision budget accordingly.
Case 3 — AI audit missed its own mistake. A translator had the LLM's translation supervised by the same LLM; The model said "no problem", a terminology error she made herself. The error was revealed when the translator cross-checked with a different model and source; The team adopted the rule that "the same model should not control itself".
Four copyable templates
1) MQM based error checking:
Your role: LQA assessor. Compare the source and translation below. Provide each error you find in the following format: [category: accuracy/terminology/fluency/style/format][weight: critical/major/minor] [location] [comment] [correction]. Mark the warning you are unsure of as "confirmation required"; errorfabrication.Source: [...] | Translation: [...]
2) Critical error scanning (high risk):
Look for critical errors ONLY in the translation below: meaning inversion, number/dose/date error, loss of negation, substitution of legal obligation, safety warning error. Ignore minor styling issues. Cite the source for each critical finding.Source: [...] | Translation: [...]
3) Automatic QA supplementary control:
List the formal inconsistencies in the following source-target pairs: number mismatch, missing/extra placeholder or tag, inconsistent term, untranslated segment, unit/currency difference. Just give the problems with their location. Pairs: [...]
4) Independent second eye (cross-check):
Evaluate this translation WITHOUT PREJUDICE; Don't assume you wrote it. Give the source a quality rating (1-5) for fidelity, terminology and naturalness, and list the top 3 problems. It's up to me to decide. Source: [...] | Translation: [...]
Weak prompt / Strong prompt
Weak: "Is this translation good?" (No criteria; the AI returns a useless answer like "generally good.")
Strong: "Check this translation against the source. Label each error with category (accuracy/terminology/fluency) and severity (critical/major/minor). Focus especially on number, negation, and terminology errors. Don't fabricate errors; mark 'confirmation needed' if you are unsure."
Difference: strong prompt gives an error frame and focus; The output is a comparable and actionable quality report.
Quality layers table
layer
what catches
Who/what does
Auto QA
Number, label, consistency
Tool/script
AI control
Possible meaning/terminology errors
LLM (with confirmation)
Human LQA
Meaning, tone, culture, weight
expert evaluator
MQM/DQF score
Objective error score
human with frame
Delivery confirmation
ultimate responsibility
competent translator
Common mistakes
- Skipping the automated QA and getting straight into reading. Stylistic errors distract attention.
- Replacing AI inspection with human LQA. AI pre-screens, does not approve.
- Having the same model check its own translation. Blind spot; cross checking required.
- Not weighting errors. Seeing critical and minor as the same disrupts the priority.
- Correcting "errors" made up by the AI without confirming them. You can distort the correct sentence.
Automatic metrics: BLEU, COMET and limits
There is also a world of automated metrics in quality measurement. BLEU (Bilingual Evaluation Understudy) compares a machine translation to a human reference translation and gives a score of 0-100 based on word overlap; It was the standard for comparing MT systems for a long time. A newer metric, COMET, is neural network based and captures semantic similarity better than BLEU. These metrics are valuable for quickly and automatically comparing two engines on big data.
But its limits are clear: Metrics like BLEU look at word overlap, not really understanding meaning. A translation that differs from the reference but is accurate may receive a low score; A translation that is similar to the reference but incorrect may receive a high score. A critical negativity error is a single word so it has little impact on the metric, but is actually catastrophic. That's why automated metrics measure trends at the system level, not decide the deliverability of an individual text. MQM-based human evaluation and expert judgment decide whether a translation goes to the client, not an automatic score. Use metrics as a compass, not a judge.
In summary
Translation quality is not a subjective feeling, it is something measurable. Automatic QA thoroughly scans for formal errors; human LQA evaluates meaning, tone, and culture; Error frameworks such as MQM quantify quality by category and weight, enabling objective comparison. AI is a powerful second eye that accelerates this control, but it can be blind to its own translation, make mistakes, and fail to fully grasp real-world weight; Therefore, the final quality decision, weighting and delivery approval belongs to the competent translator.
Application task
Get a machine translation output. List formal errors first with "automatic QA supplementary checking", then list linguistic errors by category and weight with "MQM-based error checking". I rate each error (critical 10, major 5, minor 1) with a threshold per word and ask "can it be delivered?" Answer the question. Finally, confirm with the source how many of the errors flagged by the AI are real and how many are fabricated.
checklist
- [ ] I ran automatic QA and cleaned up any formal errors.
- [ ] I marked linguistic errors with a box (category + weight).
- [ ] I scanned critical errors in a separate, prioritized round.
- [ ] I sourced the AI control and cross-checked it with a different model if necessary.
- [ ] I made the delivery decision based on the numerical threshold and my own expert judgment.