Gains:
- Ability to set up a small test set (eval) that evaluates a model with your own data
- Embed human verification, limits and responsible use policy into workflow
- Recognize hallucination, privacy and ethical risks and implement mitigating measures
You have chosen the right model; Is the job done? No, the real work starts now. Putting a model into your business comes down to testing it against your own data, validating its output, and framing it responsibly. This final unit turns the entire module into an end-to-end application: you will learn how to set up a small test suite (eval), embed human verification into the workflow, manage hallucination and privacy risks, and draw ethical boundaries. Once the unit is finished, you will be equipped to not only select a model but also commission it with confidence.
Evaluation (Eval): Testing with Your Own Data
Now you know that only your actual data, not ads, can tell if a model fits your business. The way to measure this is to set up an evaluation set (eval): a small collection of samples from your work for which the correct answer is known.
A simple eval is set up like this:
- Collect 10-30 real samples. Choose inputs that are typical of your job (mix difficult and easy).
- Determine the "correct/expected output" for each example. What would an expert answer?
- Run candidates through the same examples. Test the models you are comparing one by one with the same set.
- Score the results. In how many examples is it true? Where did he go wrong? Are mistakes acceptable?
- Compare and choose. Choose not the highest overall score, but the one that is most reliable in the most critical examples for you.
Tip: Don't blindly trust the leaderboards. A model may rank first globally, but perform worse than the second-place model in your specific Turkish legal texts. Your own eval of 20 samples is worth more than hundreds of public tests.
Hallucination: The Model's Most Insidious Risk
A hallucination is when the model produces information that is not actually true, in a confident language. Instead of saying “I don't know,” the model might produce a non-existent piece of legislation, a made-up statistic, or a false date; Moreover, he does this in an extremely convincing language. This is the most dangerous aspect of AI because error masquerades as truth.
You can't eliminate the hallucination completely, but you can reduce it and catch it:
- Base it on the source: Ask the model to generate answers from the documents you provide, not from its own "memory" (RAG logic).
- Request sources: Say, "Next to each claim, write the document/section on which it is based"; If he can't give a source, be suspicious.
- Getting people to say they're not sure: The instruction "If you're not sure, write 'not clear'" reduces model fitting.
- Human verification: Critical outputs must be verified by a human.
Caution: Never use model output without validating it for legal, medical or financial decisions. A "law article" or "dose information" produced by the model may be completely fabricated. In these areas, AI is a draft/preliminary tool; A competent person always has the last word.
Requirement for Human Verification
Artificial intelligence works best as a tool that accelerates people, not puts them out of work. The correct setup is as follows: produces or pre-qualifies the model draft; the human reviews, corrects, and approves. How stringent the verification needs to be depends on the cost of the error.
Quest
Error cost
human verification
Product description draft
low
Sample control is enough
Customer email response
medium
Pre-submission review
Contract risk analysis
high
Article by article expert confirmation
Medical/financial advice
very high
Full expert verification, AI support only
This table strikes the balance between "manually checking every output" and "not checking at all." While sample control is sufficient for low-cost jobs, every output must be verified for high-priced jobs.
Ethical and Responsible Use
Although model selection may seem like a technical decision, there are ethical responsibilities behind it:
- Transparency: Let your customers or employees know that you are using AI in appropriate places. A customer has the right to know whether they are talking to a human or AI.
- Bias: Models can inherit bias from the data on which they are trained. Monitor the fairness of outcomes in sensitive areas such as recruitment and credit assessment.
- Privacy: Embed the data protection principles you learned in unit 8 into the workflow; Do not send personal data recklessly.
- Accountability: When a mistake occurs, the "AI made it" defense is not enough. The responsibility lies with the institution using the vehicle. So document verification processes.
Tip: Write a little “responsible use policy” into your workflow: what jobs can use AI, what data can't be sent, who will verify it, and how errors will be reported. This one-page document prevents major problems in the future.
Three Realistic Cases
Case 1 — Regret without establishing eval. An insurance team begins summarizing claims without testing the most popular model. After two weeks, they realize that the model frequently makes mistakes in Turkish technical terms and reads some amounts incorrectly. When they set up an eval of 20 samples and compare the three models, they switch to the model that is not the most popular but is more accurate in Turkish technical texts. The loss: two weeks and proofreading labor. Lesson: eval first, deploy later.
Case 2 — The process that captures the hallucination. A paralegal sees a "Supreme Court decision" reference in the model printout. Since their process has a "verify every reference" rule, he looks for the decision and finds that no such decision exists; He made up a model. Thanks to human verification, fabricated information never reaches the client. Without the verification rule, the loss of reputation could have been serious.
Case 3 — Trust with transparency. An e-commerce company produces customer support responses with AI, but adds a note under each response: "This response was prepared with artificial intelligence support and was reviewed by one of our representatives." Customers are more satisfied with both the quick response and the confidence of seeing a human being engaged. Transparency is not a weakness to hide, but a source of trust.
Weak Prompt / Strong Prompt: Verifiable Output
Weak prompt:
Tell me about the risky clauses of this contract.
Can fit model; no source, no sign of uncertainty.
Powerful prompt:
Your role: contract analyst (the final decision belongs to a lawyer). Task: Highlight potentially risky clauses in the following contract. For each finding: (1) clause number and verbatim quote, (2) why it is risky, (3) your confidence level (high/medium/low). Rely only on clauses in the text; Do not fabricate anything that is not in the text. Where you are not sure, write "lawyer confirmation required". [contract]
Strong prompt links each claim to the source (article number + citation), requires confidence level, and prohibits fabrication; This makes the hallucination catchable.
Copiable Templates
1. Eval set design:
Design an evaluation set for the following task [TASK]. Suggest 15 sample inputs (mixed easy-medium-difficult), how the "expected correct output" would be defined for each, and determine a scoring criterion (1-5).
2. Hallucination-reducing wrap:
Rewrite the following prompt to reduce fabrication: "[PROMPT]". Add rules for citing sources, specifying confidence levels, and "tell me if you're not sure."
3. Verification flow design:
In the following workflow [WORKFLOW] Design a human verification step for the AI output. Determine how stringent the verification should be based on the cost of error; write who will check what, when.
4. Draft responsible use policy:
Draft a one-page "responsible AI use policy" for a team in [INDUSTRY]. Include the following headings: permitted uses, prohibited data, verification responsibility, transparency reporting, error reporting.
Common mistakes
- Deploying without Eval: Starting the model without testing it with your own data and noticing errors after they are reflected in the customer/job.
- Relying on hallucination: Using the model's confident language as truth without verifying the references.
- Bypassing human verification: Leaving output unchecked in high-cost decisions; Leaning on the “AI did it” defense.
- Avoiding transparency: Hiding your use of AI; experiencing loss of confidence when it occurs.
- Ignoring ethics and bias: Using sensitive decisions (hiring, credit) without monitoring the fairness of the output.
In summary
- Before deploying a model, set up a small eval set with your own data and test the candidates; overall rankings do not represent your business.
- Hallucination is the model's confident production of false language; Captured by source attribution, level of trust, and human verification.
- The stringency of human verification is adjusted according to the cost of error; In critical areas AI is only support, the decision is human's.
- Transparency, bias control, confidentiality and accountability; are the four pillars of responsible AI use. One page policy prevents major risks.
Application task
Set up an evaluation set of 15 examples with template 1 in this unit (eval set design) for the candidate model(s) you selected throughout the module and compare at least two models against this set. Then design how the output will be verified by humans with template 3 (validation flow) and draft a one-page policy for your team with template 4 (responsible use policy). These three deliverables turn the entire module into a workable deployment plan.
checklist
- [ ] With my own data, I can set up a small eval set and compare models.
- [ ] I can recognize hallucinations and apply mitigating and arresting measures.
- [ ] I can adjust the stringency of human verification based on the cost of error.
- [ ] I can embed transparency, bias, confidentiality and accountability principles into the workflow.
- [ ] I can draft a one-page responsible AI use policy.
Module Exam
1. What is the difference between an AI 'provider' and a 'model family'?
- A) The provider is the company that develops the model, and the model family is the model group of that company under a brand ✔
- B) They are exactly the same thing and are used interchangeably
- C) Provider is the price of the model, model family is the speed of the model.
- D) Model family refers to the company, supplier refers to a single model version
Description: The provider is the company that developed the model (e.g. Anthropic); Model family is the model group that that company collects under a brand (for example, Claude). Model version is a specific version within the family.
2. How can the 'weights' of a model be most accurately defined?
- A) It is the number of tokens that the model can produce per minute
- B) It is the billions of numerical parameters that the model learns during training and stores its information. ✔
- C) It is the monthly subscription fee of the model
- D) It is the number of languages supported by the model
Description: Weights are billions of numerical parameters that the model learns during training; It is where 'everything the model knows' is stored. The closed/explicit weight distinction depends on whether these parameters can be downloaded and run.
3. Which statement is true about 'open-weight' and 'open-source'?
- A) They are exactly the same concepts and both give unlimited commercial use
- B) Open weight always makes the training data public as well
- C) It is not the same thing; In open weighting, usually only parameters are shared and license terms may limit commercial use ✔
- D) Open source models cannot be downloaded, open weight models can be downloaded
Explanation: In open weighting, only model parameters are shared; training data and process are often not disclosed, and some licenses limit commercial use. So 'open weight' is not exactly the same as 'open source'.
4. Which of the following is the most decisive model feature for a legal team that reads and analyzes long contracts?
- A) Having the lowest token price
- B) Having visual (multimodal) processing ability
- C) Having an open weight license
- D) Having a wide context window ✔
Explanation: The model needs a large context window to process long documents as a whole; The context window is the total amount of tokens the model can keep in mind at a time.
5. Which is true about token pricing for closed weight models?
- A) The output token is usually significantly more expensive than the input token ✔
- B) Input and output are always exactly the same price
- C) Only fixed monthly fee charged, usage amount is irrelevant
- D) Input token is always many times more expensive than output
Explanation: The price is calculated per token, and the output token the model produces is usually several times more expensive than the input token you send. Therefore, long and unnecessary printouts increase costs rapidly.
6. Which feature in a model is essential for an accounting team that wants to automatically extract data from invoice images?
- A) Widest possible context window
- B) Multimodality (visual processing ability) ✔
- C) Open weight license
- D) The highest level of reasoning
Explanation: To understand visual content, the model must be able to process images as well as text, that is, it must be multimodal. Multimodality is the ability of the model to handle multiple types of data, such as text, images, and sometimes audio.
7. What is the most logical choice for a high-volume, simple task (e.g. positive/negative classification of thousands of reviews)?
- A) Always the strongest and most expensive reasoning model
- B) A model that works only on open weight and on its own server
- C) A lightweight and low-cost model, because the task is simple and the volume is high ✔
- D) The model with the widest context window
Description: A lightweight, low-cost model does the trick for simple, high-volume tasks; Using the most powerful and expensive model creates unnecessary costs here. The correct approach is to match the difficulty of the task to the model tier.
8. What triggers the sending of text containing personal data to a foreign-based AI provider in terms of KVKK?
- A) It does not create any legal liability
- B) Only matters if the provider is in the EU, in no other case
- C) It is strictly prohibited under all circumstances, regardless of whether the data is anonymous or not
- D) Data transfer abroad is considered and is subject to the special conditions of KVKK ✔
Explanation: Operating data on the servers of a provider abroad is considered 'data transfer abroad' and is subject to the special conditions of KVKK on this subject (such as explicit consent, adequacy decision, appropriate assurances).
9. What does a model's 'reasoning' mode do and how does it affect cost?
- A) It increases accuracy in complex problems, but increases cost and delay as it generates more tokens ✔
- B) Always makes the model free
- C) Only increases the number of languages supported by the model
- D) Always shorten the answers and reduce the cost
Description: Reasoning mode allows the model to 'think' step by step before giving the answer; It increases accuracy in multi-step and complex problems, but it also increases cost and delay because it generates more tokens.
10. What is the most reliable approach when evaluating (eval) a model for your own business?
- A) Just choosing the most popular model and using it without testing
- B) Set up a small test set of your own real tasks and compare models with it ✔
- C) Just looking at the benchmark score mentioned in advertisements
- D) Automatically accept the model with the highest context window as the best
Explanation: Instead of blindly relying on leaderboards, creating a small test set of your own real tasks and comparing models on this set will reveal the one that really suits your business.
11. How should the knowledge that model output may contain 'hallucinations' shape your workflow?
- A) The output should always be considered correct and published directly
- B) Hallucination is only seen in free models, not in paid models
- C) In critical decisions, the output must be verified by a human and the sources must be confirmed ✔
- D) The hallucination can be completely eliminated by simply changing the model
Explanation: A hallucination is when the model produces information that is not actually true, in a confident language. Because of this risk, verification of output by a human should be required, especially for legal, medical and financial decisions.
12. What is the basic logic of the 'model portfolio' approach adopted by mature institutions?
- A) To ensure simplicity by having every job done by a single, most expensive model.
- B) Using only the cheapest model and completely ignoring quality
- C) Do not use any models and do all the work manually
- D) Optimizing cost and reducing dependence on a single provider by using different models according to the task ✔
Description: The model portfolio is to use different models according to the task: cheap model for simple and bulky jobs, powerful model for complex jobs, open weight/regional model for confidential data. This both optimizes cost and reduces dependence on a single provider.
13. Why might country of origin and data retention policy be a criterion for model selection?
- A) Because it directly impacts regulatory, data sovereignty and privacy requirements ✔
- B) Only because it determines the response speed of the model
- C) Because it determines the file formats supported by the model
- D) Because it has no practical effect, it is just a marketing detail
Description: Country where the developer of the model is located and where/how much data is stored; It is decisive in terms of legal regulation, data sovereignty and privacy. For example, for an organization that wants to host within the EU, a regional provider or zero storage option becomes important.
14. Which best explains why looking for a 'single best model' in a task is misleading?
- A) All models actually have exactly the same capabilities
- B) Models are strong in different sizes, so 'best' depends on the job requirement ✔
- C) The most expensive model is automatically the best at every task
- D) Model selection should be made completely randomly
Description: Models are strong on different dimensions such as speed, cost, context, multimodality, privacy and reasoning. A model that is a leader on one dimension may be weak on another; so 'best' always depends on the job requirement.