Gains:
- Understand that p-hacking, HARKing, selective reporting, and the garden of forking paths produce false findings and see how artificial intelligence can accelerate these traps
- Ability to establish defenses such as pre-registration, analysis plan, multiple comparison discipline and sensitivity analysis with artificial intelligence support
- Being able to internalize the honest request design that will enable the artificial intelligence to reject 'find the meaningful result' type requests and that the final responsibility lies with the researcher.
In the previous unit we saw what p-value is and what it is not. In this unit we will look at a darker topic, the systematic traps that threaten statistical integrity. Most of these are not intentional cheating; These are insidious mechanisms that even well-intentioned researchers fall into without realizing it. And artificial intelligence (AI) makes these pitfalls both easier and faster, as it makes it possible to run dozens of analyzes in seconds. The aim of this unit is to establish the discipline that will transform AI from a "meaningful result finding machine" into an honest assistant.
Basic pitfalls
p-hacking (p-value hunting): It is trying the analysis again in different ways until a significant result (p<0.05) is obtained. Adding and removing variables, selecting subsamples, changing the definition of outliers, trying different transformations, using different outcome variables, stopping data collection when it becomes meaningful... Each attempt gives a new "chance"; If you try hard enough, one of them will turn out to be meaningful, even if it actually has no effect. The result: spurious findings that were published but not reproducible.
HARKing (Hypothesizing After the Results are Known): Making a hypothesis after seeing the results and presenting it as "I predicted it like this from the beginning". You look at the data and find the most striking relationship, then write the paper as if it were designed to test that hypothesis. This misleads the reader by making the discovery seem like confirmation.
Selective reporting (cherry-picking): Reporting only the "good" ones from dozens of analyzes and silently burying the rest. This is also known as the "file drawer problem": meaningless results remain in the drawer, making the literature systematically biased.
Garden of forking paths: Andrew Gelman's term. Even without a single intentional p-hacking, the researcher makes many reasonable choices based on the data (which variable, which threshold, which subgroup, which transformation). These investigative degrees of freedom are so numerous that you are effectively choosing the meaningful one from a multitude of analyses—even without any ill intent. The result is again a rising false positive rate.
Caution: Most of these traps are not deliberate tricks. The sum of seemingly innocent steps such as "I just tried a few more models", "it was cleaner when I removed the outlier", "the effect is clearer in this subgroup" can produce a spurious finding. Honesty is a matter of method, not intention.
Why does AI enlarge these traps?
Trying 3 models by hand takes time; YZ produces 50 model variants, 10 different conversions, 8 subgroups in minutes. This power is disastrous without an honest plan: a request like “find a meaningful relationship in this data” or “build a model that supports this hypothesis” turns the AI directly into a p-hacking engine. AI also wants to be helpful; tends to find a "meaningful" outcome that will satisfy you. That's why your prompt design is honesty's first line of defense.
Defenses: fair course of business
1. Pre-registration: It means recording your hypothesis, variables, model and tests in writing (possibly on a time-stamped platform) before performing the analysis. Thus, discovery is separated from confirmation; anything you change afterwards is tagged as "explorer".
2. Analysis plan: Even if you can't pre-record, write an analysis plan before diving into the data: primary hypothesis, primary outcome variable, main model. Establish the distinction between “main analysis” and “additional/exploratory analysis” from the beginning and maintain it in the report.
3. Report everything: Don't hide the models you've tried. In the “sensitivity analysis” (robustness check) section, show how different specifications change the result. The finding is strong only if it holds up under many plausible choices.
4. Multiple comparison discipline: If you have done too many tests, correct them (unit 6). Answer the question "How many tests have I done?" honestly.
5. Sensitivity analysis: Repeat your main finding with a different sample, different control set, and different transformation. If the result changes, do not hide it, report it.
Comparison chart: honest vs. trap
Application
trap shape
Honest equivalent
Model selection
Trying until it makes sense (p-hacking)
Pre-planned master model
hypothesis
Fitting after the fact (HARKing)
Pre-recorded, before data
Reporting
Don't just show the beautiful ones
All specifications + sensitivity
subgroup
Choosing the most striking
Predefined, correctable
data collection
Stop when it makes sense
predetermined sample
Four copyable prompts
1. Honest claim framework:
Your role: honest statistics assistant. My goal is NOT to find a meaningful result, but to find the right result. RULES:- DO NOT give me "try models until it makes sense" type suggestions,- tell me if you suggest multiple specifications they all need to be reported,- warn me of any steps that could lead to p-hacking. Help me clarify my main model first, separate discovery from confirmation.
2. Analysis plan draft:
Help me draft a preliminary analysis plan for a study: primary hypothesis, primary outcome variable, main model specification, predefined subgroups, multiple comparison strategy. Put "exploratory" and "confirmatory" analyzes in separate headings. Remind me to fill this out WITHOUT LOOKING at the data.
3. Sensitivity analysis:
My main finding is write sensitivity analysis code (R) for My goal is to SEE how solid the result is, NOT to choose the best one; show all results.
4. Self-monitoring:
I will explain my analysis process; Give me a checklist for p-hacking / HARKing / selective reporting and mark which of my steps are risky. Ask me back how many models/tests I have tried and which ones I plan to report.
Weak prompt / Strong prompt
Weak prompt:
Which variables show significant relationships in this data, find the best model.
This prompt is a direct p-hacking instruction: the AI tries dozens of combinations and presents the one that "makes the most sense." The result is almost certainly a spurious finding that cannot be replicated.
Powerful prompt:
Your role: honest assistant. The MAIN model I've predetermined is: Y ~ X + controls. Write code that predicts this model. DO NOT look for another "more meaningful" model. Also suggest a sensitivity analysis so I can see the robustness of the result, but know that I will report ALL results, not pick the best one. Don't let me write off any exploratory findings as 'confirmed'.
Difference: strong will fixes the main model, prohibits searching, and requires all results to be reported.
three mini cases
Case 1 — Subgroup hunting. One researcher found no effect in the overall sample; He asked the AI "in which subgroup is it significant?" YZ scanned the age, gender and region combinations and found "p = 0.03 in urban women aged 35-44". The article was based on this subgroup. His replication found no effect: this was a result of chance from dozens of subgroups. Lesson: selected subgroup after data is HARKing, not confirming.
Case 2 — Conversion hunting. The relationship that turns out to be meaningless in a student level form; tried it in log, square root, square and lag forms; He found p=0.048 in log-log form and reported only that. Luckily, one of the 5 forms tried crossed the threshold. The correct way was: to pre-select the form with theory and show the result of all forms in the sensitivity table. Lesson: selective reporting produces false significance.
Case 3 — How honest framing works. One team preregistered before analysis: single primary hypothesis, single main model, two predefined subgroups, Bonferroni correction. The main effect was not significant, but the team reported it honestly; The exploratory individual flagged one finding as "needing to be tested in the future." The study was reproducible and reliable. Lesson: honest planning makes even the “fail” outcome worthwhile.
Common mistakes
- Telling AI to "find meaningful results". This is straight up p-hacking; Put AI in an honest frame, asking for accuracy, not results.
- Presenting the discovery as confirmation (HARKing). It is misleading to write the hypothesis established after the data as "I predicted it from the beginning"; Label the exploratory finding.
- Just reporting good results. Show all specifications tested; Hiding sentiment analysis compromises integrity.
- Subgroup/conversion screening. Choosing the most significant of dozens of subgroups or transformations produces spurious findings; identify and fix in advance.
- Forgetting how many tests you've done. Keep track of the total number of attempts; No p-value can be interpreted honestly without knowing this number.
Tip: Before writing down a finding, ask yourself: "How many ways have I silently tried until I found this result, and am I reporting them all?" If some of the essays stay in the drawer, your report looks stronger than it is.
In summary
p-hacking, HARKing, selective reporting, and the garden of forking paths; Many are traps that operate unintentionally but produce spurious, unreproducible findings. AI accelerates these pitfalls because it can try dozens of analyzes instantly; “find meaningful results” type requests turn the AI into a p-hacking engine. Defense; separating discovery from confirmation with a pre-registration and analysis plan, reporting all specifications and sensitivity analysis, correcting multiple comparisons, and putting AI in an honest framework that demands the "correct result". The final responsibility always lies with the researcher.
Application task
Choose a research question and write a brief analysis plan before looking at the data: primary hypothesis, main model, primary outcome variable, predefined subgroups, multiple comparison strategy. Then estimate the main model. Then do a sensitivity analysis (different controls, outlier included/excluded, different function form) and show all the results in a single table. In one paragraph, evaluate which steps pose a risk of p-hacking and how your honest framework limits them.
checklist
- [ ] I wrote the analysis plan/master model before diving into the data.
- [ ] I labeled exploratory and confirmatory analyzes separately; I wasn't HARKing.
- [ ] I have reported all the specifications I have tried; I didn't just choose the beautiful one.
- [ ] I defined the subgroups and transformations beforehand, I did not scan them after the data.
- [ ] I kept track of how many tests I had run and applied the necessary correction.
- [ ] I used AI in the context of “find the truth” rather than “find it meaningful”; I took responsibility.