Gains:
- Ability to correctly define the null hypothesis, p-value, confidence interval and statistical power and use artificial intelligence in test selection and interpretation.
- Ability to distinguish what is and is not a p-value (not effect size, not probability of accuracy) and understand why multiple comparison correction is necessary
- Ability to recognize comments made by artificial intelligence that confuse meaningfulness with materiality and report them with effect size and uncertainty
The most used and most misunderstood tool of statistics is hypothesis testing. The basic idea is simple: we have a claim (e.g. “the mean of these two groups is different”) and its opposite, a null hypothesis (H0 — “there is actually no difference”). The test measures how well the data agrees with the null hypothesis. In this unit we will see how artificial intelligence (AI) is making test selection and interpretation easier, but why the pitfalls around the p-value are so common and dangerous. Let's start from the beginning: AI knows which test syntax to write; but it is your job to interpret whether the assumption of the test is satisfied and what the result really means.
What actually is the p-value?
The p-value is the probability that the outcome you observe, or a more extreme outcome, would occur by chance when the null hypothesis is true (i.e., there is actually no effect). A small p-value (0.05 is the traditional threshold) means "it would be surprising to see such data if there really was no effect"; This is considered evidence against the null hypothesis.
Now let's clarify what the p-value is not - which AI often confuses:
- The p-value is not the probability that the null hypothesis is true. p=0.03 does not mean "there is a 3% probability of no effect".
- The p-value does not indicate the size of the effect. A very small effect may yield a tiny p-value in a large sample.
- The p-value does not prove that the finding is significant or real. “Significant” is a statistical term, “significant” is a judgment.
- p>0.05 does not mean "no effect"; It means "we could not show the effect with this data." Absence of evidence is not evidence of absence.
Caution: The most common AI interpretation error is presenting the word “significant” as “significant/strong/major impact.” Statistical significance and practical significance are two different things; Always report the two separately.
Confidence interval and effect size: richer information
Instead of a single p-value, a confidence interval is much more informative: it gives the plausible range of values for your estimate. The sentence "Effect 1,200, 95% confidence interval [300, 2,100]" shows both direction, magnitude, and uncertainty; "p<0.05" just says "far from zero". Modern statistical practice uses the p-value not alone; recommends reporting with the trio of effect size + confidence interval + p-value.
Statistical power and sample
Statistical power is the probability of detecting an effect that actually exists (1 minus the probability of type II error, i.e. "missing the real effect"). An underpowered study is susceptible to two risks: (1) it misses true effects (false negative); (2) the few results that do turn out to be significant carry inflated magnitudes—the so-called winner's curse because only inflated predictions can cross the threshold. Therefore, it is good practice to calculate power/sample before analysis. AI generates the code for this account; You determine the target effect size with theory.
Multiple comparison problem
When doing a test, a threshold of 0.05 means "5% risk of false positives". But if you do 20 tests, on average one would be expected to give p<0.05 by chance, even when all of them are actually ineffective. This is called the multiple comparison problem. The probability of false positives increases rapidly when large numbers of variables, subgroups, or outcome variables are tested. The solution is to correct the threshold: Bonferroni (dividing the threshold by the number of tests — strict), Holm or Benjamini-Hochberg methods that control the false discovery rate (FDR). This risk becomes even greater in the age of AI, as AI can produce dozens of tests in seconds — this is the direct subject of the next unit (p-hacking).
Comparison table: what the p-value says and what it doesn't say
expression
Is it true?
Description
"p=0.02, probability of no effect is 2%"
wrong
p is not the probability of H0
"p<0.05, the effect is large"
wrong
p does not indicate size
"p=0.20, no effect"
wrong
Not being able to show is not absence
"p<0.05, there is evidence against H0"
True
Cautious and on point
"Effect 1.200 [300;2.100], p=0.01"
best
Size + uncertainty + evidence
Four copyable prompts
1. Choosing the right test:
Your role: statistical testing assistant. I want to compare the mean of two independent groups (continuous variable). Give me: which test is appropriate (with its assumptions: normality, equality of variance), the alternative if the assumption is not met (e.g. Welch, Mann-Whitney), and the R code. I will run the result and check the assumptions.
2. Reporting the p-value correctly:
Report the test output below, but RULES:- DO NOT present the p-value as "probability of H0" or "effect size",- be sure to include the effect size and 95% confidence interval,- write "significant" and "significant" separately,- if p>0.05, DO NOT say "no effect", say "we could not show". Output: [paste]
3. Power/sample calculation:
I'm planning an experiment: two groups, expected effect size (Cohen's d) ~0.4, power 0.80, alpha 0.05. Write R code that calculates the required sample size (pwr package) and explain the risks of a low-powered study (winner's curse).
4. Multiple comparison correction:
I've done 15 separate hypothesis tests and I have p-values. Write R code that applies the Bonferroni and Benjamin-Hochberg (FDR) corrections, explain the difference between the two methods, and summarize in one sentence why I should not trust the bare p<0.05.
Weak prompt / Strong prompt
Weak prompt:
Is the result of this test meaningful? Comment on the result.
This claim traps AI in the “meaningful/non-meaningful” binary; most likely produces a sentence that confuses magnitude with significance, such as "yes, it is significant, the effect is strong."
Powerful prompt:
Your role: attentive statistical assistant. Interpret the result but: (1) give the effect size + 95% confidence interval + p-value together, (2) separate significance from significance, (3) p>0.05 in "could not show", (4) ask how many tests I did; If more than one, suggest multiple comparison correction, (5) remind that the assumptions of the test should be checked.
Difference: strong prompt enforces rich reporting and protects against p-value traps.
three mini cases
Case 1 — Nonsignificant “significance” in large sample. One analyst found the mean difference between the two groups "highly significant" at p<0.001 in data with 500,000 observations. But the difference was only 0.3 points (out of 100) and was practically meaningless. The huge sample made even the tiny difference "significant". When the effect size was reported, the finding was found to be insignificant. Lesson: significance is not magnitude; If n is large, every difference is significant.
Case 2 — Multiple testing illusion. One team tested 22 outcome variables; One of them showed p=0.04 and was announced as "we found the effect". With Bonferroni correction, the threshold decreased to 0.05/22 = 0.0023; 0.04 did not pass this threshold. The only "meaningful" result would have been by chance out of 22 trials. Lesson: when testing a lot, correction is necessary.
Case 3 — Underpower and the winner's curse. A small pilot study (n=15 per group) found an effect very large (d=0.9) and significant. A large replication study reduced the effect to d = 0.2. The small sample had only allowed an overestimation to cross the threshold. Lesson: Significant effect sizes at low power cannot be trusted.
Common mistakes
- Confusing meaningfulness with importance/magnitude. The effect is not large just because p is small; Report effect size separately.
- Mistaking the p-value for the probability of H0. p is not the "probability of no effect"; is conceptually different.
- Reading p>0.05 as "no effect". Failure to show is not evidence of absence; State the uncertainty.
- Not correcting for multi-testing. Too many tests produce false positives; Apply Bonferroni/FDR.
- Ignoring power. The significant effect in the small sample is exaggerated; Calculate power before analysis.
Tip: Before writing off a result as “significant,” ask two questions: “Is the effect practically significant (magnitude)?” and “How many tests did I take before I found this result?” The second question is the litmus test of honesty.
In summary
Hypothesis testing and p-value are powerful but misunderstood tools. The p-value is the probability of seeing the data when the null hypothesis is true; The probability of H0 is not the magnitude of the effect or the significance of the finding. Always report significance along with the effect size and confidence interval; Do not read p>0.05 as "no effect"; apply multiple comparison correction if you have done many tests; Be aware of the risk of exaggerated effects from low-powered studies. AI produces test code and blueprint; It is your responsibility to distinguish between meaningfulness and materiality and to check assumptions.
Application task
Make a comparison of means between two groups in a data set. Ask the AI for the appropriate test (with its assumptions) and code, run it and check the assumptions. Report the result with three components: effect size, 95% confidence interval, p-value. Then think about how many different comparisons you can make on the same data; If you have run multiple tests, apply Bonferroni or FDR correction and report if the result still holds. Finally, write a rough power assessment for your analysis.
checklist
- [ ] I selected the test with its assumptions and checked the assumptions.
- [ ] I reported the result as effect size + confidence interval + p-value.
- [ ] I kept “significant” and “important” separate; I didn't think p was the probability of H0.
- [ ] I wrote p>0.05 as "we could not show it" rather than "no effect".
- [ ] I applied multiple comparison correction if I ran multiple tests.
- [ ] I evaluated the sampling power; I was cautious about the effect size at low power.