Gains:
- Ability to produce meaningful derivative features with domain knowledge and encode categorical categories with appropriate methods (one-hot, label, target)
- Ability to scale numerical variables according to model type (standardization, normalization) and avoid unnecessary or incomplete scaling
- Ability to avoid feature leakage by learning all transformations after the training/test split and only from training
There's an old saying in machine learning: "Applied machine learning is essentially feature engineering." Because the success of a model often comes from what inputs you give to the model, rather than which algorithm you choose. Feature engineering is the art of producing meaningful signals from raw data that the model can learn from. Artificial intelligence is a rich source of ideas at this stage: when you ask "what features can be produced from this data", it lists dozens of suggestions. But some of these suggestions may be valuable, some may be useless, and some may be dangerous (leak). It's your job to sort it out.
Why feature engineering
Raw data rarely comes to the model in its best form. While the "date of birth" column alone is meaningless, the "age" value generated from it is a strong signal. You can extract attributes such as "day of the week", "day/night", "is it a holiday" from the "order timestamp". You can combine two columns to produce a ratio ("debt/income ratio"). Here, feature engineering is translating domain knowledge into mathematical signal; And that's exactly why it is the stage that requires the most human intelligence.
Converting categorical variables to numbers: coding
Models generally work with numbers, not text. Converting categorical variables (such as city, color, product type) into numbers is called encoding. Three common methods:
One-hot encoding: Opens a separate column with a value of 0/1 for each category. For "city", Istanbul, Ankara, Izmir columns are formed; If a customer is from Istanbul, only that column will be 1. Ideal when the number of categories is small; If there are too many categories it produces hundreds of columns (this is called "size explosion").
Label encoding: Gives a number to each category (Istanbul=0, Ankara=1). It's simple, but it may accidentally teach the model a sequence (like Ankara > Istanbul); so it is used with caution in unordered categories.
Target encoding: Replaces each category with the average of the target variable in that category. It is very powerful, but the most dangerous source of leakage: if you take into account the target of the test data, the model sees the future. It should be calculated only from training data and carefully (within cross-validation).
Scaling: large numbers do not overwhelm the model
Some models (distance-based ones, linear models, neural networks) are sensitive to the scale of the variables. If "income" (0-500,000) and "age" (0-100) fall into the same pattern, income may dominate simply because it is larger. Scaling fixes this. Two common methods: standardization (converts each value to "how many standard deviations away from the mean") and normalization (min-max normalization — compresses values into the range 0-1). Tree-based models (decision trees, random forest) are scale insensitive, they do not require scaling.
Caution: Scaling and coding parameters (mean, standard deviation, category-mean mapping) should only be calculated from the training data, then the same should be applied to the test data. Including test data is leakage and makes your model look better than it actually is.
The heart of leakage in feature engineering
Feature generation is where data leakage most often originates. Two typical mistakes: Time leak — producing a feature that includes future information (including days after the forecast day when calculating the “last 30 day average”). Statistics leakage — calculating a feature (scaling average, target encoding value) from all the data before the training/test split. Rule: learn each transformation after doing the train/test split first and only from the training data. The safest way to do this regularly is to use pipeline — a structure that collects all transformations in a single chain and applies them after splitting.
Method
for what
Risk of leakage
note
One-hot encoding
Variable with few categories
low
Explodes size in multiple categories
Label coding
Sorted category
low
Out of order teaches wrong order
target encoding
Multi-category, strong signal
very high
Just from education, in CV
standardization
Linear/distance models
medium
Parameter depends only on education
Time window feature
time series
high
Add the future
three mini cases
Case 1 — Valuable property. A credit team generated the “debt-to-income ratio” feature from the raw “monthly income” and “monthly debt payment” columns. This single derived feature increased model accuracy from 71% to 79%; because it was the rate, not the absolute income, that really determined the risk. Lesson: ratios generated by domain knowledge are strong signals.
Case 2 — Target encoding leak. One team converted the “zip code” into a number with the target encoding (the average churn rate in that area), but did so from all the data before splitting. The model gave 94% on the test set, dropping to 68% in production. 6 weeks of effort wasted. Lesson: target coding is done carefully, only within training.
Case 3 — Scaling forgetting. One analyst fed revenue (0-400,000) and customer age (18-75) into a distance-based model without scaling. The model looked almost exclusively at income, crushing the age effect. When scaling was added, segmentation became meaningful. Lesson: scaling is not neglected in distance/linear models.
Four copyable templates
1) Feature idea generation (elimination is up to you):
Your role: feature engineering assistant. My df columns: birth_date, order_time (timestamp), income_tl, debt_tl, city, product_category. Target: "will the loan be repaid" (0/1). Suggest 15 features that can be generated from these columns; specify the risk of leakage (low/medium/high) for each. Clearly mark those that contain future information.
2) Secure coding (post-split):
Write code that one-hot encodes "city" and "product_category". IMPORTANT: fit the encoding only to the training data, then transform the test data (with sklearn OneHotEncoder). Explain how you handle the unseen category (handle_unknown) in education.
3) Leak-free conversion with Pipeline:
Set up sklearn Pipeline: apply StandardScaler to numeric columns, OneHotEncoder to categorical columns, add a classifier at the end. Guarantee all transformations are learned AFTER the train/test split and only from training. Explain the code and why it is leak free.
4) Time window feature (leakage control):
Generate the "number of orders in the last 30 days" attribute for each customer, but NEVER include data after the forecast day. Explain line by line that the code does not look into the future. I will provide the reference date column.
Weak prompt / Strong prompt
Weak prompt:
Add good properties to this data.
“Good” is undefined, the target is unclear, there is no leakage control. AI generates random, perhaps leaky, features.
Powerful prompt:
Your role: feature engineer. Target: "churn in 30 days" (0/1),estimated reference date: save_date. There is transaction history in df.Task: Generate 8 features, answer the question "Do I have this information at the time of prediction" for EACH. Adding the date after the reference date in time window features. Write the code in a pipeline-compatible way to be executed after the train/test section.
Here, the target, reference time and leakage control are defined from the beginning.
Common mistakes
- Learning the transformation from all data before division. If the scaling/encoding parameter sees the test data, leakage occurs.
- Careless use of target coding. It is the most powerful but most leaky method; just from training, in cross validation.
- Adding the future with a time window feature. If the "last 30 days" calculation is entered after the forecast day, the model sees the future.
- Unnecessary scaling in the tree model and incomplete scaling in the linear model. Scaling decisions are made according to the model type.
- Adding AI's every feature suggestion without question. Suggestions may include useless and leaky features.
Tip: Write down a single question for each feature you generate: “Can I calculate this value with the information I have at the time I make the prediction?” If the answer is not a clear "yes", do not use the feature. This single discipline eliminates most feature-related leaks.
In summary
Feature engineering is the art of generating meaningful signals from raw data and often determines the success of the model more than the algorithm. Encoding categoricals (one-hot, label, target), scaling numericals (standardization, normalization) and producing derivative features with domain knowledge are the basic tools. But this phase is also the heart of the leak: all transformations must be learned after the training/test split and only from the training data. AI generates plenty of ideas; It is human judgment that distinguishes the valuable from the dangerous.
Application task
Choose a target variable and design at least five derivative features from the columns you have. For each of them, answer the question "am I available at the time of the prediction" in writing and eliminate at least one as "high risk of leakage". Then code the secure features in a pipeline to be implemented post-partition.
checklist
- [ ] Did I apply all transformations after the train/test split?
- [ ] Did I only learn scaling/encoding parameters from training?
- [ ] Have I answered the "do I have it at the time of prediction" question for each feature?
- [ ] Have I taken extra care with high-risk methods such as target coding?
- [ ] Have I decided to scale appropriately for the model type (tree/linear)?