Gains:
- Ability to define the problem type (classification, regression, clustering) and success criteria according to the real cost of the job
- Ability to divide the data honestly (without touching the test set; chronologically in the time series) and establish a leak-free evaluation basis
- Ability to choose an interpretable model by starting with a simple baseline and adding complexity only when it deserves it.
So far we have collected data, cleaned it, explored it, and produced features. Now we come to the real work, building a model. A model is a mathematical structure that learns a pattern from data and produces predictions for new situations. But the most critical part of building a model is not the code itself, but the two decisions that precede it: defining the right problem and splitting the data correctly. AI is a powerful advisor in algorithm selection, code writing, and parameter tuning; but it is up to one to decide what to predict and what success means. An ill-defined problem won't work even with a perfectly written model.
Problem definition first: what do we predict
Every modeling job starts with a question, and this question determines the type of model. Classification — the output is a category: "Will this customer leave or stay?", "Is this transaction fake?". Regression (in English regression - the output is a number): "How much is this house worth?", "How many orders will come next month?". Clustering (dividing unlabeled data into natural groups): "How many natural segments are my customers divided into?".
The second part of the problem statement is the success criterion: what does it mean for this model to be "good"? In a fraud model, missing a fraudster is much more expensive than accidentally blocking an honest customer; Therefore, not "general accuracy" but "rate of catching fraudsters" comes to the fore. If you don't define this criterion from the beginning, together with the business owner, you'll end up with a "highly accurate" model that doesn't work (we'll go deeper into metrics in Unit 7).
Caution: "Accuracy" can be misleading. If 10 out of 1000 transactions are fraudulent, a stupid model that says "none are fraudulent" will give 99% accuracy but won't catch a single fraudster. Choose success criteria based on the real cost of the problem.
Training/test splitting: honest examination of the model
Testing a model with the data it has learned is like asking the student the same questions he/she studied on the exam; He gets high marks but doesn't show what he really knows. So we split the data into two (often three):
- Training set (training set in English, usually 70-80%): The model learns on this.
- Test set (test set, usually 20-30%): The model does not see this at all; real performance is measured here.
- Validation set: Intermediate set used for model setting (which parameter is better); to keep the test set "clean".
The most basic rule: the test set is never touched during training. Scaling, coding, feature selection — all are simply learned from the training set, then applied to testing (leakage principle from Unit 5). The test set is the first time the model sees the real world; If you turn it on too early you'll never know the true performance.
Time series exception: If your data is time dependent (sales, stock market, demand), random splitting is not done. Because random splitting causes the model to see the future and predict the past — this is leakage. Instead, divide chronologically: train with the old period, test with the new period.
Algorithm selection: from simple to complex
The most common mistake of beginners is to start with the most complex model. The correct approach is the opposite: first establish a simple baseline. "always guess the majority class" in a classification; "always estimate the mean" in a regression. This stupid model gives a baseline; If your actual model cannot pass this, there is a problem. Then move on to simple and interpretable models.
model
Problem type
strong point
weakness
Baseline (majority/average)
both
Gives benchmark
does not learn
Logistic regression
Classification
Simple, interpretable
Linear relationship only
Linear regression
regression
Simple, fast
linear assumption
decision tree
both
interpretable
Easy memorizations
random forest
both
Strong, durable
Less interpretable
Gradient boosting (XGBoost etc.)
both
very strong
Difficult to adjust, risk of memorization
Rule: choose the simplest "good enough" model. Interpretability is more valuable than power in most business contexts; It's better to explain a loan rejection "because your debt-to-income ratio is high" than "because the black box said so."
three mini cases
Case 1 — Incorrect problem definition. A team built a classification model based on "is the customer satisfied" but chose "accuracy" as the success criterion. In the data, 88% of customers were satisfied; The model got 88% accuracy by calling everyone "satisfied" and never caught the dissatisfied people — even though the real goal was to find them. Lesson: choose success criteria based on real purpose.
Case 2 — Time leak in the compartment. In a demand forecasting project, data was randomly split. The model gave 93% accuracy but crashed in production because in training it had predicted January, then November, with December data — it had seen the future. When we switched to chronological division, the actual performance increased to 74%. Lesson: chronological division in time series.
Case 3 — Unnecessary complexity. One analyst started directly with a deep neural network, tuned it for weeks, and got 81% accuracy. Then a colleague got 80% with 20 lines of logistic regression — much faster, interpretable and easier to maintain. Lesson: start with a baseline and a simple model, add complexity when it deserves it.
Four copyable templates
1) Clarifying the problem definition:
Your role: modeling consultant. Help me define this job: “I want to reduce customer churn.” Ask me and clarify: (1) is this a classification or regression, (2) what exactly is the target variable and how should it be defined, (3) what should be the success criteria and why. The decision is mine; you present the questions and options.
2) Safe training/test compartment:
Split my df into training/testing 80%/20%. Maintain class distribution (stratify).random_state=42. This is NOT a TIME SERIES (independent observations). Print the class ratio of each set after division. Just give the splitting code to the test set without applying any transformation.
3) Time series chronological division:
The data is time dependent (date column: order_date). NOT random, divide chronologically: oldest 80% training, newest 20% testing. Print out the training and testing date ranges so I can verify the future hasn't leaked.
4) Setting a baseline:
I have a classification problem (target: churn 0/1). First establish a baseline: measure training/testing accuracy with DummyClassifier, which always predicts the majority class. Then train a simple logistic regression and compare whether it beats the baseline. Show the metrics of the two side by side.
Weak prompt / Strong prompt
Weak prompt:
Build the best model with this data.
"Best" is undefined; There is no problem type, no goal, no success criteria and no division strategy. The AI generates a random pattern, possibly leaky.
Powerful prompt:
Your role: modeling assistant. Problem: classification, target "churn" (0/1), there is class imbalance (~12% churn). Success criterion: Recalling those who churn is a priority. Data independent observation (not time series). Task: (1) 80/20% stratified split, (2) DummyClassifier baseline, (3) logistic regression, all transformations in pipeline and learned only from training. Do not touch the test set before splitting.
Here the problem type, imbalance, criterion and leakage measure are clear.
Common mistakes
- Not choosing the success criteria according to the real purpose. "Accuracy" in unbalanced data is misleading; If you want to capture the minority class, recall comes to the fore.
- Touching the test set during training. Doing scaling/encoding before splitting is leaky and hides real performance.
- Random splitting in time series. The model sees the future, collapses in production; Chronological division is essential.
- Jumping into a complex model without establishing a baseline. Without benchmarks you can't know if a model is really good or not.
- Ignoring interpretability. In business decisions, a simple explainable model is often more valuable than a black box.
Tip: Before you start building a model, write one sentence: “This model will predict _____, its success will be measured by the metric _____, because the true cost of the job is _____.” If you can't fill out this sentence, you're not ready to write code yet.
In summary
The most critical part of building a model is not the code, but the decisions that precede it: defining the right problem (classification or regression, what is the goal, what is the success criterion) and splitting the data fairly (without touching the test set; chronological in the time series). Always start with a simple baseline and add complexity only when it deserves it; interpretability is valued over power in most business contexts. AI is a powerful advisor in algorithm selection and code, but the human decides what you predict and why.
Application task
Define a prediction problem and complete the following sentence in writing: “This model will predict ___ (classification/regression), the target is ___, the success criterion is ___ because ___.” Then split the data with the right strategy (chronological if it's a time series), establish a baseline, and measure whether a simple pattern breaks that baseline.
checklist
- [ ] Have I clearly defined the problem type (classification/regression) and the target?
- [ ] Have I chosen the success criteria based on the actual cost of the job?
- [ ] Did I leave the test set untouched during training?
- [ ] If it is a time series, did I divide it chronologically?
- [ ] Have I established a baseline before moving on to the complex model?