Gains:
- Ability to use artificial intelligence as a thought partner in drafting research questions, hypotheses, scale selection and analysis plans
- Ability to audit support for quantitative analysis (code/output interpretation) and qualitative analysis (thematic coding draft) with statistical and methodological accuracy
- Ability to develop a protective reflex in the use of artificial intelligence against p-value hunting, fabricated statistics and unethical data processing
Psychology is a science that systematically studies human behavior, and the basis of this science is research. Whether you are writing a thesis, measuring program effectiveness in an institution, or preparing an article; Good research requires a solid design, accurate analysis and honest interpretation. Artificial intelligence (AI) can be a thought partner at every stage of research: sharpening the research question, evoking possible scales, drafting an analysis code, explaining a statistical output in plain language, drawing initial suggestions of themes in your qualitative data. But AI also hallucinates in statistics, suggests methodological error, and most dangerously, can unwittingly open the door to unethical practices (like p-value hunting). In this unit you will learn to use AI as a powerful but supervised assistant in research.
Where does AI help in research?
Design phase: Clarifying the research question, sharpening the hypothesis statement, considering possible variables and confounders (third factors that affect the result but are not examined), reminding appropriate designs. AI is a brainstorming partner here; You decide.
Quantitative analysis: Explaining when an analysis method (such as t-test, ANOVA, regression) is appropriate, drafting analysis code (e.g. R or Python), interpreting a statistical output. AI writes code, but you are responsible for the accuracy of the result produced by the code and the suitability of the method.
Qualitative analysis: Initial code suggestions in interview transcripts, possible theme titles, codebook draft. AI gives a starting point; The researcher adds interpretive depth.
Caution: AI is not a statistician. It may suggest the wrong test, omit assumptions, produce a fudged coefficient or p-value. Verify each methodological decision and each number independently (with your own knowledge, a statistical source, or an expert).
Ethical red lines: p-hacking and data processing
One of the most dangerous pitfalls of research ethics is p-hacking: repeatedly changing the analysis, adding/removing variables, or trimming data until a meaningful result emerges. Ask AI “what can I do to make my result meaningful?” Asking " is to fall into this trap without realizing it. To this type of question, AI can give suggestions such as “extract this variable, look at this subgroup”; These suggestions "work" statistically, but are scientifically fraudulent.
Likewise, AI may suggest “completing” data that does not exist or deleting outliers without justification. Data is analyzed as it is; Changes can only be made with predefined, transparent and justified rules.
Tip: Write down your analysis plan (which test you will do, with which variables, for which hypothesis) before you see the data. Use AI to clarify that plan, not to “beautify” the outcome. A pre-written plan is the strongest protection against p-hacking.
Step by step: Safe research flow powered by AI
- Clarify the question and hypothesis. Let the AI research the problem to sharpen it, but you decide.
- Write your analysis plan in advance. Which test, which assumption, which hypothesis. Let AI oversee the plan.
- Draft code, verify output. Let AI write code; You run it, check the result independently.
- Check the assumptions. Test yourself the assumptions that AI skips, such as normality and homogeneity of variance.
- Consider qualitative codes as drafts. Consider the themes of AI initial; Read and verify the data yourself.
- Report honestly. Write down the meaningless result as well; Avoid p-hacking and selective reporting.
three mini cases
Case 1 — p-hacking trap. When a researcher's main hypothesis does not turn out to be significant (p=0.21), he asks the AI "ways to make the result meaningful." AI suggests various subgroup analyzes and variable extraction; When the researcher tries these, he gets p = 0.04. But this result is spurious: If you try 10 different analyses, one of them will be significant by chance. The right way: reporting the predefined analysis, writing the meaningless result honestly.
Case 2 — Code and output comment. A graduate student wants to test differences between two groups in data from 120 participants, but is unfamiliar with statistical software. It prints the analysis code draft to the AI, runs it, and has the AI interpret the output. Then he checks the assumptions of the t-test from a statistical source and independently verifies the result. AI shortens the learning curve, but the final decision is up to the student.
Case 3 — Qualitative coding. A researcher feeds 18 anonymous interview transcripts to the AI and asks for initial code suggestions. AI comes up with around 30 codes and 5 theme suggestions. The researcher considers these as the beginning and reads all the transcripts himself; It adds a theme that the AI missed, separating two that it incorrectly combined. It also checks reliability with a second encoder. AI stepped up, but the interpretation came out of the human.
Copiable prompts and templates
Help me sharpen a research question. My draft question: [question]. Ask/suggest the following: dependent-independent variables, possible confounders, appropriate research design. The decision is mine; Just explain the options and risks, don't force the outcome.
Check out my analysis plan: [plan]. Evaluate: does the test I have chosen fit the hypothesis, what assumptions should I check, does the sample size seem sufficient, are there any methodological risks I have overlooked? Give suggestions to solidify the plan, not to change the outcome.
Interpret the following statistical output in plain language: what result means what, what assumptions need to be checked, where is the risk of overinterpretation? State any non-significant results honestly. Output: [output]
Below are anonymous interview transcripts. In the first stage, POSSIBLE code and theme SUGGESTIONS are generated for thematic coding. Remember that these are an initial draft and I will be reading and verifying the data; Don't present it as a definitive result. Transcripts: [anonymous transcripts]
Weak prompt / Strong prompt
Weak prompt: "The result was not significant in data, how do I make the p-value significant?"
This prompt calls the AI to suggest p-hacking; Even if the output "works" statistically, it is scientific fraud.
Strong prompt: "The analysis plan I defined earlier is [the plan]. The result was not significant (p=0.21). How should I report this honestly, what could be the possible implications of the nonsignificant result, and what would I recommend for a future study?"
This prompt directs the AI to honest science; He does not distort the result, he learns from it.
Common mistakes
- Invitation to p-hacking. Asking "Make the result meaningful"; This is scientific fraud.
- Applying the method without validating it. Using the AI suggested test without checking its assumptions.
- Accepting made-up statistics. Not confirming the coefficient/p-value given by the AI with your own analysis.
- Mistaking qualitative codes for final analysis. Reporting themes of AI without reading the data.
- Hiding the meaningless result. Selective reporting by reporting only what is meaningful.
In summary
AI is a powerful thought partner in research in question sharpening, analysis plan auditing, code drafting, output interpretation, and inception of qualitative coding. But he makes mistakes in statistics and hallucinations; The most dangerous thing is that it can unknowingly open the door to unethical methods such as p-hacking. Write out your analysis plan in advance, verify each method and number independently, check assumptions, treat qualitative codes as drafts and read the data yourself, and report meaningless results honestly. AI accelerates; Scientific integrity comes from you.
Application task
Set up a small research scenario (e.g. comparing two groups on a scale). First, write your analysis plan in one paragraph. Then have the AI audit this plan and ask if there are any assumptions or risks you missed. Also, have the AI interpret a small imaginary output and check for yourself whether its interpretation is overly assertive. Write an honest reporting sentence without engaging in P-hacking.
checklist
- [ ] I wrote my analysis plan before seeing the data.
- [ ] I have independently checked the assumptions of the method suggested by AI.
- [ ] I verified every statistic the AI gave with my own analysis.
- [ ] I considered the qualitative codes as initial drafts and read the data myself.
- [ ] I also reported insignificant results honestly.
- [ ] I avoided p-hacking and selective reporting.