Gains:
- Ability to choose the test appropriate to the type and distribution of the data and check the assumptions (normality, variance)
- Ability to understand the true meaning of the p-value and report the results with effect size and confidence interval
- Protection from p-hacking with discovery-verification separation, multiple comparison correction and full reporting
The experiment is over, the data is in. Now we are at the most critical and most mistaken stage: analyzing the data correctly and interpreting the results honestly. Biological data is often noisy, unstable and multivariate. Incorrect test selection, implicit assumption violations, and unconscious p-hacking (trying an analysis until it gives the desired result) are the primary causes of the reproducibility crisis in the published biological literature. AI helps you choose the right test, check assumptions, and write analysis code; but statistical integrity is your responsibility.
In this unit, we will cover common statistical tests, assumption checks, and ways to protect yourself from p-hacking.
Choosing the right test
The choice of test depends on the type and distribution of the data. Artificial intelligence will help you make this choice, but you shouldn't apply it blindly:
- Two groups, continuous data, normal distribution: Independent t-test.
- Two groups, non-normal: Mann-Whitney U (non-parametric: not requiring distribution assumption).
- More than two groups: ANOVA (analysis of variance) + post-hoc test.
- Categorical data (counts): Chi-square or Fisher exact test.
- Relationship: Correlation (Pearson/Spearman) or regression.
Tip: Visualize the data before selecting the test. A histogram or boxplot instantly shows the normality and outliers that a t-test requires. "Graph first, test later" is a good habit.
Checking assumptions
Every test has hidden assumptions. The t-test assumes normal distribution and equality of variance; If these are violated, the result will be misleading. AI writes code that checks these assumptions:
Role: You are a biostatistics assistant. Task: Write code that checks the assumptions of a t-test before comparing two groups of data: normality (Shapiro-Wilk), equality of variance (Levene). If the assumption is violated, tell me which alternative test I should move on. Give Python code with comments.
The code redirects you to Mann-Whitney if normality is broken. This “check first, decide later” flow keeps you from performing the wrong test.
p-hacking and how to protect yourself
p-hacking is analyzing data in different ways until p<0.05: trying different tests, selectively discarding outliers, adding/removing groups, reporting what is "significant" among a large number of variables. This is the main source of fake discoveries. Ways of protection:
- Fix the analysis plan in advance (Unit 7).
- Report all tests, not just the ones that show significance.
- Fix multiple comparison (FDR/Bonferroni).
- Give the effect size and confidence interval next to the p-value.
- Separate discovery and validation: Test what you find in the discovery set on an independent set.
Step by step: an honest analysis
- Visualize the data (scatter, outlier).
- Check the assumptions.
- Perform the test you predetermined.
- Fix multiple comparison.
- Report effect size + confidence interval.
- Write the limitations clearly.
Copiable prompt templates
Visualize throughput: plot histograms and boxplots for each group, flag outliers. Then interpret normality.Data columns: [name]. Give Python code with comments.
I want to compare my two groups. Characteristics of the data: [type, N, distribution].Which test is appropriate? Check assumptions, perform the test, report effect size AND 95% confidence interval WITH p-value.
I tested for 15 different genes in my analysis. Give the code that applies the Bonferroni and Benjamini-Hochberg correction and explain the difference between the two methods.
Weak prompt / Strong prompt
Weak: "Are these two groups different, do a t-test."
Strong: "I have two groups (control n=18, treatment n=20), the dependent variable is enzyme activity (continuous). First visualize the distribution and test normality with Shapiro-Wilk. If normal, apply t-test, if not, Mann-Whitney. Report the result with p-value, effect size (Cohen's d or rank-biserial) and 95% confidence interval. Explain which test you chose and why."
Difference: The powerful prompt has data type, sample numbers, assumption checking and full reporting request. The model follows the right path instead of blindly applying t-test.
three mini cases
Case 1 — Wrong test: A student applied a t-test to significantly skewed (non-normal) data and found p=0.048. When the artificial intelligence added the normality check, the data did not come out normal; With Mann-Whitney, p=0.11. The actual result was meaningless. Lesson: testing without checking the assumption is misleading.
Case 2 — Selective outlier discard: A researcher deleted two “outlier” points to make the result meaningful. The model emphasized that outlier discard should be based on a predefined rule (e.g., IQR method) and reported; arbitrary deletion is p-hacking. Lesson: set the outlier rule beforehand.
Case 3 — Effect size recovery: One study had p=0.001 but the effect size (d=0.1) was almost zero; the large sample showed the insignificant difference to be significant. When the artificial intelligence reported the effect size, the finding was found to have no practical value. Lesson: Just because p is small doesn't matter.
comparison chart
Status
right approach
To be avoided
abnormal data
Nonparametric test
Forced t-test
multi test
Apply correction
Crude p<0.05
outlier
Predefined rule
arbitrary deletion
significant result
Effect size + CI
just p
discovery finding
Validate on independent set
Finalize in one set
Common mistakes
- Hypothesis uncontrolled testing: Spurious significance with false testing.
- p-hacking: Trying analysis until it gives the desired result.
- Reporting only p-value: Omitting effect size and confidence interval.
- Not correcting for multiple comparisons: False positives.
- Causation jumping: Inferring causality from correlation.
Caution: AI will not "produce" the result you want, but will happily write the code for the wrong test if misled. You have statistical integrity: report all trials, plan the analysis in advance, never miss the effect size. Work with a biostatistician for analyzes leading to publication.
real meaning of p-value
The most misunderstood concept in biology is the p-value. The p-value is not the "probability of the hypothesis being true"; It is "the probability of seeing a difference as big or more extreme than what you observe, when in reality there is no difference." This subtle but critical distinction is key to not exaggerating the results. p=0.03 does not mean "the effect is 97% real". When having the AI interpret a result, check that it uses this definition correctly; The model can sometimes repeat common misinterpretation.
The confidence interval (CI) is often more informative than the p-value because it shows the magnitude and uncertainty of the effect together. “Difference 2.4-fold (95% CI: 1.8-3.1)” says much more than “p<0.05”: both the direction of the effect, its magnitude, and how precisely it was measured. Highlight GA in your reports.
Use the correct definition of the p-value when interpreting my result. Avoid incorrect statements such as "probability of the effect being true." Instead, explain the practical meaning in terms of the effect size and 95% confidence interval. Result: [data]
Correlation, causation and observational data
Most biological data is observational: you see something changing with something else, but you can't see what causes what. The sentence “expression of gene X correlates with disease” does not indicate that X causes disease; The disease may be increasing X, or a third factor may be affecting both. Causality can only be established through interventional experimentation (changing X and measuring the result). AI may unknowingly use causal language (“causes,” “leads to”) in your texts; review each conclusion sentence from this perspective, expressing observational findings in correlational language (“correlated,” “varying together”).
Separating paired and independent data
The most frequently overlooked distinction in test selection is whether the samples are independent or paired. If you are taking before and after measurements from the same individual (for example, blood values of the same patients before and after treatment), these measurements are not independent; A paired test is required. Using the independent two-sample t-test here discards match information in the data and reduces power; It may even reverse the outcome. The rule is simple: "Do these two measurements come from the same biological unit?" If the answer is yes, paired test (paired t-test or Wilcoxon signed rank test) is used, if no, independent test is used. When you ask artificial intelligence for tests, be sure to specify the match structure of your data; If you do not specify, the model often assumes independence and recommends the wrong test.
I have two sets of measurements. Important: these measurements were taken before/after the SAME individuals (paired). Choose the right test that takes into account this match (paired t-test or Wilcoxon), check your assumptions and report with the effect size. Don't assume independence.
In summary
Accurate data analysis; choosing the appropriate test for the type of data, checking assumptions, correcting multiple comparisons, and reporting the effect size and confidence interval in addition to the p-value. The biggest danger is p-hacking; The antidote is an advance analysis plan, full reporting, and discovery-verification separation. Artificial intelligence is a powerful aid in test selection and assumption checking, but statistical integrity lies with the human.
Application task
Take a data set (your own data or a sample) with two groups. First have the AI write code that visualizes the data and tests for normality. Depending on the result, apply the appropriate test (t-test or Mann-Whitney) and report the effect size and confidence interval along with the p-value. Then compare the effect of multiple comparisons without correction and with correction on the result.
checklist
- [ ] I visualized the data before testing.
- [ ] I checked the test assumptions (normality, variance).
- [ ] I chose the appropriate test, I did not blindly apply the t-test.
- [ ] I fixed multiple comparison.
- I reported the [ ] p-value as well as the effect size and confidence interval.
- [ ] I fixed the analysis plan in advance and avoided p-hacking.