Gains:
- Ability to choose the metric appropriate to the problem type and business context (PR-AUC/recall in unbalanced data, MAE/RMSE in regression) and recognize over/under learning
- Ability to interpret each metric against a baseline and measure deviation and separate noise from progress with cross-validation
- Ability to report non-optimistic results by tuning hyperparameters with the validation set and using the test set only at the end
Model training (training: the process of learning patterns from data) is the most visible but most misleading step of ML engineering. It appears because it produces a satisfactory number like "accuracy 95%". Misleading because that number is often the right answer to the wrong question. In this unit, education and evaluation are discussed with the engineering discipline; We use artificial intelligence as a partner in experimental design and base the decision on measurement.
Logic of the training cycle
A model learns the pattern in the data by minimizing a loss function: a function that numerically measures the model's error. The optimization algorithm (e.g. gradient descent) reduces the loss by adjusting the parameters step by step. The aim is not to memorize the training data, but to generalize it to unprecedented data.
Two main dangers:
- Overfitting: The model memorizes the training data and fails on new data. Training success is high, verification success is low.
- Underfitting: The model cannot capture the pattern; Both training and validation success are low.
Finding the balance is the art of education. The validation set is there to monitor this balance: if validation success begins to decrease while training success increases, overlearning has begun.
Tip: Plot the training and validation loss together at each step. The point at which the two curves begin to diverge is where overlearning begins and is the right moment to "early stop".
Metric selection: the most critical decision
The wrong metric makes a good model look bad and a bad model look good. The problem determines the metric:
- Unbalanced classification (one class is very rare, e.g. fraud): Accuracy is misleading. A model that says "call everything normal" will get 99% accuracy but won't catch a single fraud. Instead, precision (how much of what I caught is actually positive), recall (how much of what I caught is true positive) and their balance F1 or PR-AUC are used.
- Balanced classification: Accuracy and ROC-AUC may be appropriate.
- Regression (number estimation): MAE (mean absolute error), RMSE (penalizes large errors), MAPE (percentage error).
- Ranking/recommendation: NDCG, MRR, Recall@K.
Whether precision or recall is important depends on the business context. Recall (not missing any patients) is a priority in cancer screening; Precision in spam filter (not sending important e-mails to spam) is important. This is a business decision, not a technical one, and is made together by the engineer and the owner.
Weak prompt / Strong prompt
Weak prompt: "Evaluate my model's performance, accuracy 0.97."
Powerful prompt: "I have a fraud detection model; positive class rate is 1.5%. Accuracy is reported as 0.97. Explain why this metric might be misleading, tell me which metrics (precision, recall, PR-AUC) I should prefer and why. Also calculate how accurate a 'call everything negative' baseline model would get on this data, so I can see the real added value."
Difference: powerful prompt gives class ratio and business context; it also asks for a baseline model comparison — this is the most important anchor for whether a metric is meaningful.
Baseline: metric without comparison is meaningless
A metric is not good or bad by itself; It is good or bad according to a basic model. The basic model is the simplest solution that comes to mind: "always tell the majority class", "repeat last week's value", "guess the mean". If your model can't clearly pass this simple solution, all the complexity is for nothing.
Caution: The sentence "My model is 85% accurate" by itself does not say anything. If the base model already gets 84%, your model is almost worthless; If the base model gets 50%, your model is perfect. Always speak in terms of the basic model.
Cross-validation and trust
A single training/testing split may be due to chance. Cross-validation: dividing the data into k parts and testing each part sequentially shows how stable the performance is. In 5-fold cross validation you get five different scores; Their mean and standard deviation are important. If the average is 80% but the deviation is ±12%, your model is unstable — it may behave very differently in the next batch of data.
This is also critical for the "two model comparison". If Model A got 81% and Model B got 82%, is B really better? If the deviation is ±3%, this difference may be noise. Consider whether the difference is significant before deciding.
Hyperparameter tuning: with validation, not testing
Hyperparameters (settings manually determined before training—learning rate, tree depth, etc.) are set with the validation set. The test set is used only at the end, once. If you select hyperparameters by looking at the test set, the test set will be contaminated and the performance you report will be optimistic which is not the case in reality.
Artificial intelligence is a good helper in designing the hyperparameter search space and writing the search code (grid search, random search, Bayesian optimization). But you still decide "which metric will we optimize?"
three mini cases
Case 1 - Accuracy trap. A medical team was proud of a model that detected a rare disease: 98% accuracy. When the basic model comparison was made, the truth emerged: since the disease rate was 2%, the model that said "call everyone healthy" also received 98%. The model's recall was only 11% — missing most patients. Once the metric was converted to PR-AUC, the actual performance was measured and the model was redesigned.
Case 2 - Mistaking noise for progress. A team spent months improving the model from 86.2% to 86.9%. Cross-validation showed that the bias was ±1.4% — so the 0.7 point “improvement” was statistical noise. The team had wasted three weeks on unreal earnings. Lesson: don't declare victory without verifying that the improvement is greater than the deviation.
Case 3 - Contamination of the test set. An engineer repeatedly looked at the test set to choose the best hyperparameters. The 91% it reported dropped to 83% in production. Why: he had unknowingly selected the model accordingly by looking at the test set over and over again (overlearning on the test set). Reported and actual performance overlapped when a separate validation set was used.
Copiable templates
Help me choose the right metric for this classification problem. Problem: [what is predicted] Class distribution: [positive rate
Evaluate the comparison of the following two models.Model A cross-validation: [list of scores]Model B cross-validation: [list of scores]Calculate the mean and standard deviation. Is the difference statistically significant or is it just noise within the deviation? Which one would you recommend I choose and why?
Check this training code for the following:1) Are the hyperparameters selected with the test set or validation set?2) Is early stopping monitored with the correct set?3) Are there any signs of overlearning (training/validation loss difference)?Code: [code]
Which of the MAE, RMSE, and MAPE should I report for this regression problem?Scale of the target variable: [range]Are large errors disproportionately bad (RMSE), or are they all equal (MAE)?Are there values close to zero (do they distort MAPE)?Suggest with brief justification.
Metric selection table
Problem type
Appropriate metric
To be avoided
Why
Unbalanced classification
PR-AUC, F1, recall
Accuracy
Majority class inflates the metric
Balanced classification
Accuracy, ROC-AUC
—
Reliable in balanced data
Regression (significant outlier)
RMSE
MAPE (if zero)
Punishes major mistakes
Regression (equal weight)
MAE
—
Easy to interpret
Ranking/recommendation
NDCG, Recall@K
Accuracy
Order is important
Common mistakes
- Using accuracy on unbalanced data. The most common metric error.
- Not making base model comparisons. It makes the metric lose its meaning.
- Looking at the test set for hyperparameters. Optimistic, unrealistic result.
- Relying on a single compartment. Without cross-validation you won't see the bias.
- Mistaking noise for progress. Small "improvements" from deviation are mostly luck.
- Ignoring the business context. The precision/recall balance is a business decision.
In summary
The real skill in model training is not to produce a high number, but to know what that number means. The problem and business context determine the right metric; interpret each metric according to a baseline model; measure bias with cross-validation and don't mistake noise for progress; Use the test set only at the end, one time. AI is your partner in experiment design, but the decision of “good enough” is up to you and yours.
Application task
For a classification model: (1) write the class distribution, (2) build an appropriate base model and measure its score, (3) evaluate your model with 5-fold cross validation and report the mean and standard deviation, (4) calculate the metric appropriate to the problem (e.g. PR-AUC) instead of accuracy. Write down whether your model beats the baseline model by a large margin without bias.
checklist
- [ ] I chose the metric based on the problem type and business context.
- [ ] I built a base model and compared against it.
- [ ] I reported the mean and deviation with cross-validation.
- [ ] I confirmed that the improvement is greater than the deviation.
- [ ] I selected the hyperparameters with the validation set.
- [ ] I only used the test set once, at the very end.