Gains:
- Ability to select and interpret classification and regression metrics (confusion matrix, precision/recall/F1, MAE/RMSE/R²) according to the purpose of the job
- Ability to detect overlearning by comparing training and test scores and obtain reliable performance with cross-validation
- Ability to evaluate whether each model adds real value by comparing it with a baseline
It's easy to set up a model; It is difficult to honestly measure whether it actually works. The subject of this unit is a data scientist's most critical skill: evaluating the model with the right metrics, achieving reliable predictive performance, and catching the most insidious trap: overlearning (memorization). AI calculates, interprets and compares metrics; but it's up to the human to decide which metric is right for your business and whether a model is "good enough". A team looking at the wrong metric can mistake a bad model for months as a “success.”
Why accuracy is not enough: confusion matrix
The first metric that comes to mind in classification is accuracy (accuracy — the total ratio of correct predictions). But as we saw in Unit 6, accuracy is misleading with unbalanced data. A better start is the confusion matrix — a table that divides predictions into four bins:
- True positive (TP): We called fake what is really fake. (Good)
- True negative (TN): We said really clean. (Good)
- False positive (FP): We mistakenly said that the clean one was fake. (False alarm)
- False negative (FN): We missed the fake and called it clean. (Missed threat)
Two key metrics derive from these four boxes. Precision: "How much of what I call fake is actually fake?" — important if false alarm costs are high. Sensitivity (recall in English): "How many real fakes did I catch?" — important when missing a threat is expensive. The two are often at odds with each other: if you lower the threshold and say "fake" more, recall increases but precision decreases. The F1 score is the balanced average (harmonic mean) of these two.
Attention: The answer to the question "Which metric is important" is a business decision, not a technical one. Missing a case (low recall) in cancer screening is disastrous; It is annoying to send an important email to spam (low precision) in the spam filter. Cost determines the metric.
Regression metrics
If the output is numbers, different metrics are used. MAE (Mean Absolute Error): how much the estimates deviate from the actual average, in the same unit (e.g. "we are wrong by 4,200 TL on average"). RMSE (Root Mean Squared Error): penalizes major errors more severely and is sensitive to outliers. R² (R-squared — coefficient of determination): how much of the variability in the target the model explains (close to 1 is good, 0 is as bad as predicting the mean, negative is even worse).
The most insidious trap: overlearning
Overfitting — where the model memorizes training data but fails on new data — is the most common mistake in data science. The symptom is clear: the model is great on the training set (98%), bad on the test set (72%). The model has memorized the noise of that particular sample, not the actual pattern in the data. There is also the opposite: underfitting—the model is bad in both training and testing because it is too simple to capture the pattern.
Ways to combat overlearning: simplifying the model, more data, regularization — the mathematical brake that keeps the model from getting too complex — and most importantly, cross-validation.
Cross-validation: don't trust single pane
A single training/test pod may be lucky or unlucky; You can get high marks for the test set just because that sample is easy. Cross-validation (CV for short) solves this. The most common form is k-fold CV: data is divided into k parts (e.g. 5); In each round, one part is testing and the rest is training; this rolls 5 times and the 5 scores are averaged. So you'll see that performance is based on the average of multiple splits, not a single lucky split. The distribution of scores is also informative: very volatile scores (85% on one floor, 62% on the other) indicate that the model is unstable.
Normal k-fold is not used in time series (leaks the future); Instead, you do "forward chaining" (time series split): always training with the past and testing the future.
metric
Problem type
When does it matter?
Accuracy
Classification
If classes are balanced
Precision
Classification
False alarm is expensive
Sensitivity (recall)
Classification
If it's expensive to miss
F1
Classification
If balance is needed
MAE
regression
Interpretable error
RMSE
regression
If big mistakes are critical
R²
regression
explanatory power
three mini cases
Case 1 — The illusion of high accuracy. One fraud model showed 99.2% accuracy and the team celebrated. Looking at the confusion matrix, the real: fake rate in the data was 0.8%; the model caught almost no fakes, just saying "all clear". Recall was 6%. Lesson: look at the metrics, not the accuracy.
Case 2 — Latent overlearning. One team delivered and boosted XGBoost to 97% on the training set. The test set was 69%, but no one had looked. The model collapsed in production. If cross-validation had been done, the large difference between folds (overlearning) would have been visible from the beginning. Lesson: trust CV and test score, not education score.
Case 3 — Lucky split. One analyst was delighted to get 88% in a single split. When his colleague did a 5-fold CV, the scores were 88%, 71%, 83%, 64%, 79% — the average is 77%, but it's very volatile. The model was unstable; The single compartment was misleading. Lesson: look at the CV mean and distribution, not the individual bin.
Four copyable templates
1) Full classification report:
I have trained model and test set. Generate: confusion matrix, precision, recall, F1 (for each class) and support counts. I'm inferring, but it's up to me: I'll tell you which metric is important. Note that accuracy can be misleading with unbalanced data.
2) Overlearning control:
Print my model's scores in both the TRAINING and TEST sets side by side. Calculate the difference between them and warn as a large difference is a sign of overlearning. If the difference is large, suggest regularization or simplification.
3) Cross validation:
Perform 5-fold cross-validation (for stratified, classification). Print the score, mean and standard deviation of each fold. If the scores are very volatile (high std), indicate that the model is unstable. Use pipelines so that transformations are learned only from the training piece at each layer.
4) Baseline comparison:
Put my model's metric side by side with a DummyClassifier baseline. Does the model significantly beat the baseline? If it doesn't pass, make it clear that the model doesn't add any real value.
Weak prompt / Strong prompt
Weak prompt:
Is my model good?
"Good" is undefined; It is not clear which metric, which threshold, which baseline. AI can give a single accuracy number and mislead.
Powerful prompt:
Your role: assessment assistant. Classification, unbalanced data (12% positive). Priority: recall (not missing the positives). Task: (1) confusion matrix,(2) precision/recall/F1, (3) 5-fold stratified CV mean and std,(4) DummyClassifier baseline comparison. Focus on recall and baseline difference, not accuracy. Comment, but I make the final decision.
Here, the metric priority, CV and baseline condition are clear.
Common mistakes
- Trusting accuracy in unbalanced data. 99% accuracy may not capture the minority class at all; Look at the confusion matrix.
- Just looking at your education score. High education, low test score is overlearning; Always compare two scores.
- Relying on a single compartment. Lucky division is misleading; Look at the mean and distribution with cross-validation.
- Not comparing with baseline. You can't say "good" without knowing whether the model beats a stupid predictor.
- Using normal k-fold on time series. It leaks the future; time series split is required.
Tip: Write three numbers next to each model: training score, cross-validation average, and baseline score. This trio reveals at a glance overlearning (training >> CV) and undervaluing (CV ≈ baseline).
In summary
Building a model is easy, but evaluating it honestly is difficult. In classification, confusion matrix, precision, and recall are much more informative than accuracy; The cost of the job determines which one is important. MAE, RMSE and R² are used in regression. The most insidious trap is overlearning: always compare training and test score. Rely on the mean and distribution of cross-validation, not a single split; Compare everything to a baseline. AI calculates all of these, but it's up to the human to decide which metric to use.
Application task
Extract confusion matrix, precision, recall and F1 for a classification model; Then do 5-fold cross-validation and look at the score distribution across folds. Finally, write down the training score, CV average, and a baseline score side by side. Comment in one sentence whether your model is overlearning and passing the baseline.
checklist
- [ ] Have I looked at the actual metric of the job (precision/recall/F1 etc.) instead of accuracy?
- [ ] Have I examined the confusion matrix?
- [ ] Have I compared training and test score and checked for overlearning?
- [ ] Have I looked at the mean and distribution with cross-validation?
- [ ] Have I compared the model to a baseline?