Gains:
- Ability to make research traceable by recording methodological decisions, prompts and model information
- Ability to separate raw data from processed data and document all conversion steps
- Being able to transparently declare the role of artificial intelligence in the method section and apply the 'can it be reproducible' test
The most fundamental feature of science is that a finding is not based on the word of a single person, but on a process that can be verified by others. Reproducibility is the ability of another researcher, taking the data and method of a study, to reach the same result by following the same steps. A similar concept is transparency: the researcher clearly documents what he did and why he made which decision. In the age of AI, these two concepts have become both easier and more critical. Easy, because AI helps document processes; critical, because if you leave no trace of the AI's decisions, no one — not even you — can reproduce your work. In this unit you will learn how to make your research reproducible and transparent.
The heart of the problem is this: When you work with AI, you make lots of little decisions — which prompt you used, which model, which settings, which output you accepted and which you rejected. If these decisions are not documented, you may get a different output when you run the same prompt on a different day (language models do not give the exact same response every time). When a reviewer or colleague asks "how did you arrive at this theme analysis", "the AI said so" is not a scientific answer. Transparency is precisely about opening this “black box” and making the process traceable.
Step by step: establishing a traceable research
1. Keep a decision diary. Write down every important methodological decision—why this sample, why this code, why you accepted this AI proposal—in a dated journal. AI helps organize this log.
2. Save prompts and model information. Keep a record of every prompt you used, which AI model you ran it on, and on what date. This is the “materials list” for your work.
3. Separate raw data from processed data. Keep the original data unchanged; do all conversions (cleaning, encoding) separately with documented steps. So everyone can watch from the beginning.
4. Write clearly the role of AI. In the method section, state honestly in which tasks and how you used AI. Most journals and institutions now require this.
5. Share data and code if possible. When ethics and privacy allow, sharing anonymized data and analytics code in open repositories is the pinnacle of transparency.
Tip: A simple test: "Can I do this research myself again in 6 months?" If you haven't documented your prompts, decisions, and steps, the answer is most likely "no." Make it producible for yourself; Then it happens for someone else too.
three mini cases
Case 1 — Traceless analysis collapsed. One researcher performed a theme analysis with AI but did not record which prompt he used. When the referee asked "how did you arrive at these themes?" he could not show the process; He had to repeat the analysis all over again, this time keeping a diary. He lost two weeks.
Case 2 — Decision log ends discussion. A disagreement arose in a team asking "why did we combine these codes?" But the decision log contained the date and justification for each merger decision. The argument was resolved within minutes when the AI summarized the log and displayed it chronologically.
Case 3 — Transparent disclosure built trust. In one article, the authors explicitly wrote that we used AI “for transcript editing and initial code suggestion, with two researchers manually verifying all codes.” Reviewers found this transparency positive; It was found to be much more reliable than using a cloaked AI.
Four copyable templates
1) Decision log template:
Create a methodological decision log table. Columns: date, decision, rationale, alternatives, role of AI (if any), who approved it. Insert my decision notes below into this table and mark any missing rationales as "must be completed". Notes: [here]
2) Draft AI use statement:
Draft a paragraph that transparently explains my use of AI to put in the methods section. I used it in these tasks: [list]. I did NOT use it in the following tasks: [list]. I verified all outputs manually. Model and date: [x]. Use honest, understated, academic language.
3) Reproducibility inspection:
I describe my research process: [summary]. Can another researcher reproduce this? List undocumented, unclear, or untraceable steps: which data, which transformation, which prompt, which decision is missing? Tell me what to record for each space.
4) Data provenance map:
List all the transformation steps from raw data to my final finding as a flow: raw data → cleaning → coding → analysis → finding. Show what was done at each step, with what tool and with what decision. Mark the untrackable step. Process: [here]
Weak prompt / Strong prompt
Weak prompt:
Write the method of this analysis in one paragraph.
AI produces a generic method paragraph without documenting the process; How the decision was made and where AI was used remains unclear. This is not manufacturable.
Powerful prompt:
Write a producible description for the method section. Use the following information: data source [x], sample [y], analysis steps [z], the AI model I used and its date [t], the exact role of the AI [r], the human verification process [d]. Write concretely enough that someone else can follow these steps and reproduce them. Ask me for any information I left out.
The difference: the second approach makes the process concrete, traceable and truly reproducible.
Transparency layers
layer
What is documented
Why is it important?
decision diary
Methodological decisions + rationale
defends the judiciary
Prompt registration
Prompt used, model, date
Makes AI tractable
Data origin
Raw → processed conversions
opens steps
AI statement
The role and limit of AI
honesty
sharing
Anonymous data + code
Ultimate transparency
Common mistakes
- Not saving prompts. If you don't keep the instruction you used, you can't demonstrate the process.
- It means "The AI said so." Scientific justification is process and judgment, not the word of the model.
- Overwriting the raw data. Keep the original; Make conversions separate and documented.
- Hiding the use of AI. Transparent statement builds trust; Hiding causes loss of reputation.
- Leaving it unproducible even for yourself. A job that you cannot do again after 6 months is not science.
In summary
Reproducibility and transparency transform a study from one person's word into a science that anyone can test. This is even more critical when working with AI, because if the model's decisions are not followed, the process becomes a black box. Keep a decision log, record prompts and model information, separate raw data from processed, honestly declare the AI's role, and share data and code when ethics allow. A study that passes the "Can I reproduce myself after 6 months" test is also reliable for others.
Application task
For a small analysis you are doing or planning, record at least 5 methodological decisions with justifications with the "decision journal template". Keep the prompts you use, model and date in a separate file. Then use the “reproducibility audit” template to have the AI find undocumented gaps in your process and close each gap. Finally, write an outline of the "AI use statement" and a transparent paragraph to be placed in the methods section.
checklist
- [ ] I wrote the reasons for each important methodological decision in the journal.
- [ ] I recorded the prompts I used, the model and the date.
- [ ] I kept the raw data unchanged and documented the transformations.
- [ ] I declared the role of AI honestly in the method section.
- [ ] I applied the "Can I reproduce after 6 months" test.