Gains:
- Ability to establish schema and rule-based output validation layers
- Ability to meaningfully require human-in-the-loop in high-impact decisions
- Ability to design verification and trust threshold based routing with the second model
A language model produces fluid, persuasive, and often accurate—but “persuasive” is not the same as “correct.” The model can silently fit an amount, a date, or a JSON field; This is called hallucination (the model confidently produces information that does not exist in reality). In an enterprise system, if that output flows to the next step — a payment, an email, a database write — the error spills over into the real world. In this unit, we will learn to filter the output with verification layers before it enters the system and to require human-in-the-loop in high-impact decisions.
Why Is Output Validation Required?
Model output can be corrupted in two primary ways: format (does not conform to the expected JSON schema, field is missing/excess) and content (the format is correct but the value is wrong — a non-existent product code, an illogical date). There is a third dimension in terms of security: malicious output (a malicious command produced as a result of injection or leak). A solid system stops all three at the door.
Caution: "Model generally accurate" is not a production criterion. In a system without verification, even one error in a thousand means 100 erroneous transactions per day in 100,000 requests per day.
Layers of Authentication: Step by Step
- Schema validation. Check with the machine that the output conforms to the expected structure: are the fields present, are their types correct, are the required fields filled?
- Rule/business logic validation. Do the values match the business rules? (Amount > 0, date is not in the future, product code belongs to the catalogue.)
- Reference/source control. If the model produces an assertion, can it be linked to the source? (Is the RAG quote actually in the document?)
- Validation with the second model (LLM-as-judge). An independent model evaluates the output as "correct/incomplete/risky".
- Trust threshold and orientation. If the model or validator reports low confidence, the output does not automatically pass; is directed to humans.
- Human control. A high-potency or low-safe outcome depends on an expert's approval.
Four Copiable Templates
Scheme + "make it up if you don't know" together:
Return the response in the following JSON schema ONLY: Write "low". NEVER write an estimate as if it were exact.
Verification with second model (judge prompt):
You are an independent validator. Below is a <source> text and a <claim>. Check to see if EVERY number and date in the claim occurs verbatim in the source. For each, say: "verified | not in source | contradicts source." If even one of them is 'absent/conflicting', mark the result as "HUMAN REVIEW REQUIRED".<source>{{ text }}</source><claim>{{ model_output }}</claim>
Trust threshold routing rule:
Routing rule:- emin_misin = "high" AND amount < 10,000 TL -> automatic processing- emin_misin = "medium" OR amount 10,000-100,000 TL -> second model verification- emin_misin = "low" OR amount > 100,000 TL -> human approval required
Human audit summary card (speeds up review):
When presenting the decision to a person, produce this card:- What is being proposed? (one sentence)- What source is it based on? (article/document reference)- What are the 2 weakest assumptions?- If approved, can they be reversed? (yes/no)
Weak Prompt / Strong Prompt
poor approach
Strong approach
"Subtract amount from invoice" (free text)
Strict JSON schema + null + trust field
Writing the output directly to the payment system
Schema → rule → human approval (if necessary)
Just telling the model "be sure"
Number/date validation with second model
Processing every output with equal confidence
Routing based on influence and trust
The strong approach does not hope that the model is correct; It creates a door that will catch you when you're wrong.
Three Mini Cases
Case 1 — The scheme alone was not enough. An accounting automation was extracting the amount from the invoices as JSON. The scheme was correct, but the model produced "125,000" instead of "1,250.00" on an invoice (decimal shift). The scheme failed to capture this; rule verification ("the amount must be in line with the total of invoice items by ±1%") was caught and incorrect recording of 112,500 TL was prevented.
Case 2 — The second model captured the hallucination. “30 days notice of termination,” a legal support assistant said in the contract summary; However, in the contract it was 90 days. When the independent judge flagged the model as "conflicting with the source", the output was forwarded to the human and corrected. If it were automatic, the customer would notify the cancellation based on the wrong date.
Case 3 — Routing reduced the load by 70%. An insurance claims system automatically approved low-amount and high-security claims and sent only the above-threshold/low-secure ones to the expert. Of the 3,200 daily demands, only 950 fell to humans; experts devoted their time to the truly risky 30%, with average transaction time dropping from 4 hours to 40 minutes.
Tip: Don't set up human control so that "people can see everything" — this will tire people out and approval will become a rubber stamp. Instead, only route high-impact and low-confidence outputs to the human; This focuses attention on what really matters.
Making Human Control Meaningful
Human-in-the-loop is not about putting a checkbox on paper. The reviewer must have (1) the context to understand the decision, (2) access to the source, and (3) the authority to say “no.” Otherwise, the control remains cosmetic. The review card (fourth template above) is meant to provide just that context.
Common mistakes
- Just doing schema validation and skipping content/value errors.
- Thinking that by telling the model "make sure" you are doing real verification.
- Automatically implement high-impact, irreversible decisions.
- Putting human control on every output and turning approval into a meaningless rubber stamp.
- Saying “approve” to the reviewer without giving the source and context.
- Processing all outputs with the same risk without establishing a trust threshold and routing.
In summary
- The output is corrupted in three ways: form, content, and malicious intent; a solid system stops all three at the door.
- Layers: schema validation, rule/business logic, source control, second model (LLM-as-judge), and trust threshold routing.
- Human-in-the-loop should be mandatory for high-impact and low-safety outputs.
- Human review must be meaningful: the reviewer must have context, resource access, and the authority to say “no.”
- Both safety and efficiency are gained by directing only the risky ones to humans, not every output.
Application task
Take an example from your own AI output. First define a JSON schema and force the output to it. Then write at least two business rules (for example, “amount matches total of items”). Finally, set up a routing table: which trust/influence combination goes automatically, which goes to the second model, which goes to the human? Generate a faulty sample and observe where each layer captures it.
checklist
- [ ] I define a strict schema for the output and verify it with the machine.
- [ ] I added at least one business/rules validation (value logic).
- [ ] I can link the assertions to the source and check them.
- [ ] Second model or human validation available for high impact/low safety outcomes.
- [ ] Routing rule defined based on trust and influence.
- [ ] The reviewer is provided with context, source, and authority to reject.