Gains:
- Ability to set up a data pipeline (collection, validation, cleansing, transforming, splitting, versioning) and placing schema validation at the beginning of the pipeline
- Ability to make missing value and labeling decisions based on field meaning and division to prevent data leakage (group and temporal)
- Ability to create a reproducible data base by fixing the data version and randomness seed
The real power of every machine learning system lies in the data, not the model. Experienced engineers know: “garbage in, garbage out” — even the most advanced model fed bad data will produce bad results. In this unit, we establish the data pipeline (data pipeline: the chain of steps that make the raw data ready for model training) end to end and learn at which step of this line we can safely use artificial intelligence.
Steps of data line
A data line typically passes through these stops:
- Collection (ingestion): Pulling data from sources (database, API, log files, event streams).
- Validation: Checking whether the data conforms to the expected schema, types, and ranges.
- Cleaning: Handling missing values, duplicate records, outliers, and inconsistencies.
- Transformation: Turning raw data into attributes — such as converting a categorical variable to a number, producing a "day of the week" from a date.
- Splitting: Separating into training, validation and testing sets.
- Versioning: Recording which model was trained with which data.
Artificial intelligence saves time by generating code drafts and ideas, especially in steps 2, 3 and 4. But decisions such as which record to discard, which missing value to fill and how, belong to the engineer who knows the data; because improper cleaning can inject a hidden bias into the model.
Data verification: early defense of the line
The most expensive errors begin not in production, but where the verification step is skipped. Schema validation automatically checks whether each incoming batch of data conforms to the expected structure. For example, is the age column between 0-120, is the email field empty, has the number of columns changed?
Tip: Put the verification at the beginning of the line. The sooner corrupt data is caught, the cheaper it is to fix. A schema error caught in production is many times more expensive than one caught in the training phase.
Write a validation scheme with pandera (or Great Expectations) for the following data schema. Columns and rules:- user_id: integer, cannot be null, unique- age: integer, cannot be from 0-120- signup_date: date, cannot be in the future- country: categorical, from the set {TR, DE, US, UK}- balance: decimal, cannot be negative Produce a meaningful error message for each rule violation. Show the test with an example broken line at the end of the code.
Cleaning: it is the human who decides
Missing values are a reality of every data set. Ways to handle:
- Deletion: Discarding a row/column with a very high missing rate. But there is a risk of information loss and bias.
- Imputation: Imputation with mean, median, most frequent value, or model-based prediction.
- Flag: Storing “was missing” information in a separate flag column — sometimes the missing itself is the signal.
Which one is correct depends on the problem. In a medical data set, the "blood value not measured" information should be preserved rather than deleted; Because even the doctor's refusal to take measurements is a signal. AI can give you options and code; You choose which one fits the reality of the field.
Weak prompt / Strong prompt
Weak prompt: "Fill in missing values."
Strong prompt: "There are missing values in the following columns: income (12% missing, right-skewed distribution), last_login (30% missing). Suggest filling income with median, but explain why median and not mean. For last_login, assume the missing value might be significant (user may have never logged in); consider generating a never_logged_in flag instead of deletion. Write down the bias either approach would add to the model."
Difference: strong prompt gives distribution information and area meaning; artificial intelligence produces decision support instead of mechanical filling.
Labeling: quality is measured
In supervised learning (learning in which examples are given with the correct answers), what the model learns are labels (labels: the correct answer for each example). Label quality sets a ceiling — if people label inconsistently, the model learns inconsistently.
Inter-annotator agreement measures the rate at which different people give the same label to the same sample; It is expressed by a coefficient such as Cohen's Kappa. Low compliance indicates either the task is unclear or the instruction is weak.
Artificial intelligence helps in labeling in two ways: (1) drafting the annotation guideline, (2) pre-labeling and having the human only correct it. But pre-labeling with LLM has a pitfall: systematic error of the model can leak into the entire label set. That's why humans always check some of the LLM labels.
Attention: Do not consider labels produced by LLM as "ground truth". Check a sample with a human and measure LLM-human fit. If compliance is low, pre-labeling will do more harm than good.
Data partition: prevent leakage
The most dangerous mistake when splitting data into training/validation/testing is data leakage: the mixing of test information into training. Examples:
- The same user's records fall into both training and testing (group leak).
- Using the future in training and the past in testing in time series (temporal leakage).
- Calculating scaling (normalization) parameters from all data and then dividing.
Temporal splitting is essential for problems involving time: train with the past, test in the future. Random splitting gives a "future" benefit that will never happen in production and inflates the metrics.
Data versioning and reproducibility
“What data did we train this model with?” Being able to answer the question months later is the hallmark of serious ML engineering. Data versioning stores each data snapshot with an ID (hash or version tag). Tools such as DVC (Data Version Control) version data like code.
To reproduce the result of a model, three things must be fixed: the data version, the code version, and the random seed. It is not possible to say "I got the same result" without this trio. We will deepen Reproducibility in unit 11; but fixing the seed in the data pipeline starts from here.
three mini cases
Case 1 - The day schema validation saved. When a team converted an upstream system price field from pennies to liras, all prices dropped 100 times. Schema validation rejected the batch as "price out of range" and the model was not trained with corrupted data. Without verification, the error would only be noticed in production, with incorrect predictions.
Case 2 - Bias of incorrect filling. In a credit model, missing income values were filled with the mean. But missing incomes were predominantly in the low-income group; averaging artificially "enriched" this group, and the model offered them an unfairly high limit. Fixed the problem with median + missingness flag.
Case 3 - Temporal leakage. A demand forecasting model looked great on the test set (95% accuracy) but crashed in production. Why: because of random splitting, the model had seen the future. Switching to temporal binning dropped test accuracy to 78% — but that was real performance and kept it in production.
Copiable templates
Split the following dataset into three sets: training/validation/testing.Constraint: This is a time series; Use TEMPORAL splitting (train in the past, test in the future). Prevent batch leakage: have the same `customer_id` only in one cluster. Calculate scaling parameters ONLY from the training set, then apply to all. Print how many lines are left in the code at each step and add an assert that checks for no leaks.
Write a draft annotation guideline for this labeling task.Task: [e.g. Label customer review positive/negative/neutral]Clarify borderline cases: sarcasm, mixed emotion, how to label review unrelated to product?Give 5 examples and 3 difficult edge cases that will increase consistency across taggers.
Produce a reproducibility checklist for this data pipeline:- How should the data version be fixed?- Which randomness seeds should be set where?- What metadata (data hash, row count, date) should be logged? My codebase: [language/library]
Check this cleanup code for data leakage. Specifically look at this: are the scaling/encoding parameters calculated BEFORE splitting? Are any statistics calculated from all data or just training? Code: [code]
Decision table: missing value strategy
Status
Recommended approach
Why
Numerical, skewed distribution
fill with median
The average is affected by outliers
Numerical, symmetrical
fill with average
Protects information
The deficiency may be significant
Flag column + fill
Lack is a signal
Missing rate > 60%
Evaluate/discard column
Noise is too much
Categorical
"Unknown" category
Does not create artificial majority
Common mistakes
- Skipping verification. Without schema control, corrupted data sneaks in silently.
- Scaling before splitting. It leaks test statistics into education.
- Using random splitting in time series. It produces fake high metrics.
- Blindly trusting LLM labels. Systematic error spreads throughout the data.
- Not saving the data version. You cannot reproduce the result.
- Mechanical filling with average. It ignores field meaning, adds bias.
In summary
The data pipeline is the foundation of the ML system and deserves more effort than the model. Put verification at the top; make cleaning and labeling decisions with domain knowledge; prevent leakage (group and temporal) in the compartment; fix the data version and seed. AI generates code and ideas on this line, but it's up to you to decide which data to process and how — because every wrong decision here passes into the model as a hidden flaw.
Application task
Write a validation scheme (pandera/Great Expectations) on your own dataset and deliberately add a bad row and show that it got caught. Then split the data temporally or batchwise, calculate scaling parameters from training only, and verify there is no leakage with an assert. Write the data version and row count to a metadata file.
checklist
- [ ] Schema validation runs at the top of the line.
- [ ] I chose the missing value strategy based on field meaning, I did not fill it mechanically.
- [ ] I measured label quality (compliance); I human-checked LLM tags.
- [ ] I prevented group and temporal leakage in the pane.
- [ ] Scaling/encoding calculated from the training set only.
- [ ] Data version, number of rows and seed recorded.