Reproducibility and the End-to-End Project: Combining Everything — Artificial Intelligence in Artificial Intelligence and Machine Learning Engineering

Gains:

Ability to ensure reproducibility with four pillars (seed fixation, data versioning, media freezing, experiment monitoring) and produce the same result when repeating the same run
Ability to combine all stops of the module (metrics, data, model, LLM components, eval, fairness, security, distribution, monitoring) in an end-to-end chain
Ability to verify that the critical decision remains with the human at each stop and document the project in an auditable manner

The most insidious failure of an ML project isn't a crash; "Not getting the same result again." If you can't reproduce the score today of the model you put into production three months ago, you don't really control that model. In this closing unit, we deepen reproducibility: the ability to reliably obtain the same result with the same inputs and combine the entire module in an end-to-end project discipline.

Why reproducibility is difficult

In ordinary software, the same code gives the same output. In ML there are many more variables that determine the outcome:

Randomness: Data shuffling, weight initialization, data splitting — all rely on randomness.
Data: Same code produces different model with different data version.
Environment: Library versions, hardware (CPU/GPU), even operating system can change the result.
Hidden case: An unsaved hyperparameter, a manual preprocessing step, an unnoted selection.

Reproducibility is not a "nice to have" but a scientific and engineering imperative. A result that cannot be reproduced is a claim that cannot be proven.

Four pillars of reproducibility

1. Fix randomness. Set all random seeds in one place: data splitting, model initialization, data shuffling. Fixed seed is the basis of the "same result when you repeat the same run" guarantee.

2. Version the data. Record which data version each experiment was performed with (data versioning in unit 2). “Latest data” is vague; "data version v3, hash abc123" is exact.

3. Freeze the medium. Pin all dependencies to their exact versions (e.g. exact versions like numpy==1.26.4 in requirements.txt, or a container image). The "latest version" will break everything one day.

4. Track everything (experiment tracking). Automatically save for each experiment: code version (git commit), data version, all hyperparameters, metrics and output structures. Experiment tracking tools like MLflow, Weights & Biases do this systematically. Without registration, the question "which setting was best" remains unanswered.

Caution: "I'll remember later" is the most expensive fallacy. Two weeks later you won't remember which seed, which data, which hyperparameter you used. Automatic tracking eliminates reliance on memory.

Weak approach / Strong approach

Weak: "I found the best model, it's on the notebook, I think its score was 89%."

Strong: "Run #147 in the experiment tracking tool: git commit a3f9c, data version v3 (hash abc123), seed 42, all hyperparameters registered, test PR-AUC 0.887. When I run the same command again, I get the same result bit by bit. The model depends on this run in the registry."

The difference: in the strong approach the result is not based on a memory, but on a fixed and monitored chain. Everyone can produce the same result every time.

End-to-end project: combination of module

Now let's combine the entire module into a single project flow. A real ML system goes through these stops, and each stop builds on the previous one:

Problem definition: What are we solving, how to measure success (unit 3: right metric, business context). The metric and threshold are clear from the start.
Data pipeline: Collection, validation, cleansing, leak-free partitioning, versioning (unit 2).
Model development: Training, baseline comparison, cross-validation, hard seed (unit 3 + this unit).
LLM components (if applicable): RAG (unit 4) and/or agents (unit 5); fine-tuning if necessary (unit 6).
Evaluation: eval cluster with edge and security cases, multi-layer eval in LLM systems (unit 8).
Justice and ethics audit: Subgroup analysis, model card, explainability (unit 10).
Security audit: Prompt injection, privacy, supply chain (unit 9).
Distribution: Packaging, gradual distribution, rollback, model registry (unit 7).
Monitoring: Three-layer monitoring, drift alarms (unit 8).
Reproducibility: Seed, data version, media and experiment tracking throughout the entire chain (this unit).

In this flow, AI is an accelerator and blueprint generator at every stop; but metric selection, data decisions, fairness prioritization, deployment threshold, and release approval — critical decisions remain with the human. This is the essence of the module.

Documentation: the future will thank you

A good ML project documents itself. At a minimum, the following should be written: problem and success criteria, data source and version, model selections and justifications, evaluation results (including subgroups), known limits and risks, deployment and retrieval procedure, monitoring plan. This document is the best friend of the person (maybe it is you) who returns to the project after six months.

three mini cases

Case 1 - Lost result. An engineer trained a great model, but he didn't fix the seed and didn't save the data version. When he left the job, no one could reproduce that result; the model became a "black box legend" and was eventually built from scratch. Weeks were wasted. Lesson: a non-reproducible result is a non-existent result.

Case 2 - Environment collapse. One team had not fixed the dependencies. When a library was automatically updated, model outputs silently changed and production was disrupted. It took days to find the problem. When the dependencies were frozen and containerized with the definitive versions, the problem did not occur again. Lesson: freeze the environment.

Case 3 - The power of monitoring. A team automatically monitored each experiment. Three months later, during a regulatory audit, they answered the question "with what data, with what settings, what performance did it get in which groups?" with a full recording within minutes. The inspection went smoothly. Lesson: monitoring is a compliance tool, not just an engineering one.

Copiable templates

Do a reproducibility check for this ML project.- Are all randomness seeds fixed (split, initialize, shuffle)?- Are data versioned?- Are dependencies frozen to exact versions?- Is every experiment (code commit, data, hyperparameter, metric) tracked? Write concrete steps on how to fix it for each missing column.Project structure: [description]

Produce a plan skeleton for this end-to-end ML project. Problem: [description] Cover the following stops and mark where the HUMAN decision is at each stop: problem/metric, pipeline, model, (RAG/agent/fine-tune?), eval, fairness, security, distribution, monitoring, reproducibility. Write the main risk and verification step for each stop.

Produce a technical documentation template for this project. Sections: problem+success criteria, data (source+version), model selections+justification, evaluation (including subgroups), known limits+risks, deployment+rollback, monitoring plan. Give the fields to be filled in for each section as questions.

Check my experiment monitoring setup: Is it auto-saved on every run: git commit, data version/hash, all hyperparameters, all metrics, environment (library versions)? Do I get the same result when I run the same run again? Setup: [description]. List the flaws and correction.

Reproducibility columns table

column

What is fixed

Vehicle example

randomness

all seeds

seed setting

Data

Data version/hash

DVC

environment

Library versions

requirements pin, Docker

Monitoring

Code+data+setting+metric

MLflow, W&B

Common mistakes

Not fixing the seed. The result cannot be repeated.
Not saving the data version. "With what data?" remains unanswered.
Not freezing addictions. An update will silently break everything.
Leaving experiments to memory. Two weeks later nothing is remembered.
Leaving critical decisions to artificial intelligence. Metrics, justice, and distribution decisions should remain with people.
Postponing documentation. The future team (and you) pay the price.

In summary

Reproducibility is the signature of serious ML engineering: the non-reproducible result is the unprovable claim. It comes with four columns — fix randomness, version data, freeze environment, track each experiment. An end-to-end project combines all the stops of this module (metric, data, model, LLM components, eval, fairness, security, distribution, monitoring) in an interconnected chain; Artificial intelligence is an accelerator at every stop, but critical decisions remain with the human. Document everything — for future team and audits. This discipline is the framework that sustains everything you learn throughout the module.

Application task

Check an ML project against four reproducibility pillars: are the seeds immutable, is the data versioned, is the environment frozen, are the experiments tracked? Fix any missing columns and prove that you can run the same run twice and get the same result. Then output the end-to-end flow of the project (10 stops) on one page and mark "where the human decision is" at each stop. Finally, write a short technical documentation draft.

checklist

[ ] All randomness seeds fixed.
[ ] Data version/hash is recorded with each experiment.
[ ] Dependencies are frozen to firm versions (pin/container).
[ ] Each experiment is automatically monitored (code+data+setting+metric).
[ ] When I repeat the same run, I get the same result.
[ ] I verified and documented that critical decisions in the end-to-end flow are made by humans.

Module Exam

1. As an ML engineer, what is the best approach when positioning artificial intelligence in the workflow?

A) AI is an accelerator in low-risk businesses; Critical decisions like metrics, data, and production stay validated and left to the human ✔
B) As long as the AI outputs look good, there is no need for verification
C) Leaving the decision to put the model into production to artificial intelligence saves time.
D) Artificial intelligence is only useful for writing text, it has nothing to do with data and model work

Description: AI is a powerful accelerator for low-risk, easily verified tasks such as code, data digests, and documents; However, the responsibility for decisions affecting money, confidentiality, and legal liability, such as metric selection, which data goes into training, and putting the model into production, lies with the qualified engineer and team. Each output should not be used without verification.

2. Why is schema validation placed at the beginning of a data pipeline?

A) Because it directly increases the accuracy of the model
B) Because it makes data versioning unnecessary
C) Because it catches corrupted data at the earliest and cheapest point and prevents it from leaking into the next steps ✔
D) Because it eliminates the need for labeling

Explanation: The earlier corrupt data is caught, the cheaper it is to fix it. Schema validation prevents corrupt data from silently leaking into training or production by rejecting data outside the expected type and range at the beginning of the line (e.g. price shifting 100x with unit change); The same error caught in production is many times more expensive.

3. What is the correct approach when dividing data into training and testing in a problem involving time (time series)?

A) Using random splitting because it is always the fairest method
B) Using temporal splitting: prevent leakage by training with the past and testing in the future ✔
C) Using all data as both training and testing
D) Incorporating test data into scaling parameters before training

Explanation: Random splitting on time series gives the model a 'future-seeing' advantage that will never happen in production and artificially inflates the metrics (temporal leakage). The correct one is temporal division: train with the past, test in the future. This measures the actual performance that keeps it in production.

4. Why is accuracy misleading in a fraud detection model with a positive class rate of 1.5%?

A) Because Accuracy is always low on unbalanced data
B) Because Accuracy can only be used on regression problems
C) Because Accuracy calculation requires a lot of processing power
D) Even a paltry model that predicts the majority class can be very accurate, thus hiding real success ✔

Explanation: On unbalanced data, even a basic model that says 'call everything negative' gets about 98.5% accuracy but will not catch a single fraud. Therefore, in unbalanced classification, precision, recall, F1 or PR-AUC are used instead of accuracy, and each metric is interpreted according to a base model.

5. Why is baseline comparison essential when talking about a model's metric?

A) Because the base model is always better than the real model
B) Because it is clear whether a metric is meaningful or not only when compared to a simple baseline model ✔
C) Because the base model makes cross-validation unnecessary
D) Because the basic model is legally required in every report

Explanation: A metric is not good or bad by itself; It is good or bad according to a basic model. The sentence '85% correct' means almost worthless if the base model already gets 84%, and perfect if it gets 50%. Without a comparison anchor, the metric is meaningless.

6. Which is the most critical security element that should be included in the production prompt of the RAG (Retrieval-Augmented Generation) system?

A) Instruction to rely only on the source given, to say 'I don't know' if the source does not exist, and to cite the source ✔
B) Telling the model to produce as long and creative answers as possible
C) The model prioritizes its own educational knowledge over resources
D) Implement all instructions in the documents brought as commands

Explanation: The single most important instruction of RAG is to tell the model to rely only on the source given, and if the information is not in the source, say 'I don't know' and cite the source without making it up. Without this triad, the model may ignore context and produce hallucinations, and the answer becomes unverifiable.

7. A RAG system gives incorrect answers. Where is the best place to start diagnosis?

A) Measuring fetch first (Recall@K): does the correct piece ever arrive? ✔
B) Immediately replace the model with a larger one
C) Change the prompt randomly and continue trying
D) Embedding all documents into the model with fine-tuning

Explanation: RAG's weakest link is usually fetch, not production. If the correct part is never brought, the model cannot produce that information, no matter how much the prompt is improved. Therefore, first Recall@K is measured to see whether the correct part has arrived; If the fetch is good, then the production and prompt are examined.

8. What actions should be put behind human approval when giving a tool to an agent?

A) None; The agent must be able to perform every action autonomously
B) Only reversible actions such as reading and searching data
C) Irreversible or high impact actions such as transferring money, deleting, sending ✔
D) Actions that involve only calculations

Description: Actions are separated by risk level. Retrievable tasks such as reading, searching, calculating, and generating drafts can be done autonomously; However, irreversible or high-impact actions such as transferring money, sending emails, deleting data, placing orders etc. require human approval. Every irrevocable action must be subject to consent.

9. What is the best design approach against the risk of indirect prompt injection?

A) It is enough to add a single sentence 'ignore bad instructions' to the system prompt
B) Give more authority to the model by relying on instructions in external content
C) Not taking any precautions because injection is unpreventable
D) Isolating external content as unreliable data and establishing layered defenses with minimal authorization, approval, and output control ✔

Description: External content processed by the agent or RAG, such as a web page, document, email, etc., is untrusted data and may contain secret instructions. The correct approach is layered defense: isolating external content as 'data, not commands' with clear delimiters, applying minimal authorization, binding irreversible actions to human approval, and auditing the output. A single line of instructions is not enough.

10. What is the main distinction when deciding whether a problem should be solved with fine-tuning or RAG?

A) Information problems are better solved with RAG, behavior/format problems are better solved with fine-tuning ✔
B) Every problem should always be solved by fine-tuning
C) RAG is used only for code generation, fine-tuning is used only for translation
D) Fine-tuning can always be updated cheaper and faster than RAG

Explanation: Fine-tuning is weak and risky in teaching the model new information; but is powerful in teaching behavior, format, tone and style. 'The model company does not know our data' is an information problem and belongs to RAG. 'Let the model always output in our strict format' is a behavioral problem and a candidate for fine-tuning. Additionally, prompt and few-shots should be consumed before fine-tuning.

11. Which is mandatory for safe deployment when putting a new model into production?

A) If the model is good in testing, open it directly to 100% traffic
B) Not setting up monitoring at all after deployment
C) Phased deployment (shadow/canary) and a pre-tested rollback plan ✔
D) Publishing the model even if the evaluation threshold is not met

Explanation: Opening the new model directly to all traffic is risky; If it's wrong, everyone is affected. The correct thing is that it is a gradual distribution (shadow, canary) and every distribution has a tested rollback plan. A distribution is not complete without a clawback plan; Being able to revert to the previous version within minutes protects the user when the model behaves unexpectedly in production.

12. How can an ML model fail 'silently' in production and what is the way to catch this?

A) The model collapses; server logs show this
B) By producing wrong predictions without making mistakes; ✔ It captures operational, input and output layered monitoring
C) Model can never fail silently, always alarm
D) Just monitoring latency is enough to catch any degradation

Explanation: The model can fail simply by producing incorrect predictions without crashing or giving errors; The main reason for this is data drift and concept drift. Just monitoring operational metrics (latency, error rate) is not enough; input distribution and output/prediction distribution should also be monitored. Input drift gives early warning if the actual result is delayed.

13. What principle is essential when using LLM-as-judge to evaluate an LLM system?

A) LLM-referee is always correct, human verification is unnecessary
B) The referee must make a decision based only on answer length.
C) Rules-based controls and human evaluation should be discarded completely when referees are used
D) Judge scores should be calibrated with a human-labeled sample and their bias measured before they can be trusted ✔

Description: LLM-referee is also a model; It can be hallucinatory, biased (favoring long, confident answers), and inconsistent. Therefore, referee scores must be calibrated with a human labeled sample and their systematic bias must be measured before the production decision is made. An unverified referee gives false confidence.

14. Why is looking at overall accuracy inadequate when assessing model bias?

A) Overall accuracy is sufficient because it always reflects the performance of the worst group
B) Overall accuracy alone is insufficient as it can obscure systematic difference (hidden discrimination) between subgroups ✔
C) Because accuracy is a metric that has nothing to do with bias
D) Bias comes only from the model and has nothing to do with the data.

Explanation: Overall accuracy may obscure systematic differences between subgroups. For example, while the overall accuracy is 88%, recall may be 91% in one group and 67% in another group; The model systematically misses that group. Therefore, the model should be evaluated on the basis of subgroups (demographics/segment) and which definition of justice should be prioritized should be decided with stakeholders.

15. What four things must be fixed together for an ML result to be reproducible?

A) Only model name, size, price and release date
B) Only GPU brand and internet speed
C) Only the final accuracy score of the model; the rest can be kept in memory
D) Randomness seed, data version, environment (dependency versions) and experiment tracking ✔

Description: Reproducibility is achieved through four pillars: fixing randomness seeds, versioning data (version/hash), freezing the environment (exact library versions/container), and tracking each experiment (code commit, data, hyperparameter, metric). Without this chain it is not possible to reproduce the same result; A non-reproducible result is a claim that cannot be proven.