Gains:
- Ability to recognize in which forms (fabricated gene/sequence/variant/reference) the hallucination (artificial intelligence's self-confident production of errors) occurs in genetics
- Ability to understand that the AI's 'confident' tone is not evidence of accuracy and cross-verify each output with an independent source
- Ability to implement a hallucination-reducing workflow with source enforcement, coordinate/version confirmation, and 'if you don't know, make it up' instructions
One danger recurs in each unit of this module: the hallucination of artificial intelligence (AI) — that is, producing with complete confidence information that does not actually exist. This unit addresses that danger on its own: what and how does AI fit in molecular biology and genetics; How do you notice this? and what verification reflex you should develop for each type of output. The goal is for you to see the hallucination not as an "accident that sometimes happens" but as a phenomenon that is always expected and systematically captured.
Why are hallucinations inevitable? Because LLM is not a system that "knows the truth", but a system that "produces the next most likely word". It has learned what a gene name, a variant form, or a tag pattern looks like in the training data; Therefore, it can produce output that is very similar to reality but not reality. In genetics, this similarity is particularly dangerous because a made-up "c.1234A>G" and a true variant cannot be distinguished by eye.
What does AI make up in genetics? a map
made up thing
What does it look like
How to catch
Gene name/symbol
Realistic but non-existent symbol
HGNC/NCBI Gene
series
Wrong sequence with correct letters
NCBI/Ensembl FASTA
Variant coordinate
In HGVS format but wrong/non-existent
VariantValidator, ClinVar
population frequency
A reasonable percentage
gnomAD
functional work
persuasive imprint
PubMed/DOI
clinical claim
"It is pathogenic"
ACMG chain of evidence
protein coordinate
3D numbers with decimal
PDB/AlphaFold file
Statistics result
p-value without correction
Rerun the code
Techniques that reduce (but do not end) hallucinations
1. Give context. The more accurate context you give (genome version, transcript, actual sequence), the less the AI will make up. But it doesn't reset.
2. "If you don't know, tell me" instruction. The instruction "If unsure, say 'must be verified', don't make it up" reduces fabrication — but don't trust it completely.
3. Require the resource. Request verifiable source (DOI, PMID, database record) for each claim.
4. Get it self-checked. “What elements in this output require independent verification?” ask; AI can often flag risky items.
5. Most importantly: verify in person. The above reduces the probability; Only your primary source control provides certainty.
Tip: Set a “validation budget”: with each AI output, be sure to verify the facts critical to the decision (sequence, variant, frequency, attribution); Treat low-stakes explanations (definition of a concept) more loosely. Focus your resources on the mistake that can cause the most harm.
three mini cases
Case 1 — Nonexistent gene. A student tried to research a "gene" that AI mentioned but couldn't find it in HGNC. The AI had produced a symbol that did not exist by combining the names of two real genes. Without HGNC control, the student would spend hours searching for a ghost gene.
Case 2 — Chain hallucination. A researcher asked the AI about a variant; The AI gave a made-up frequency, then suggested a false ACMG code based on that frequency, then added a fake article supporting that code. A single false value gave birth to a three-layered false chain. When the researcher opened gnomAD, the first link in the chain broke and everything collapsed.
Case 3 — Confirmation gained. A technician received a string analysis code from the AI; In the code, the AI had embedded a made-up string as a "reference string". The technician read the code and replaced the sequence with the actual sequence from NCBI. If he ran the code blindly, the result would be silently wrong.
Confirmation reflexes: by type of output
When you see SEQUENCE -> when you see NCBI/Ensembl FASTA when you see VARIANT -> when you see VariantValidator + ClinVar + gnomADFREQUENCY -> search in gnomAD itself when you see ATTRIBUTE -> open DOI/PMID, keep title-author When you see GENE NAME -> when you see HGNC / NCBI GeneCOORDINATE -> when you see genome version + liftOverSTATISTICS -> run the code yourself, check the correction
Four copyable templates
1) Self-control prompt:
In the output you just gave, which elements (sequence, variant, frequency, gene name, citation, number) require independent verification? Label each one as "certain/possible/unsure" and write down which source to check.
2) Source mandatory response:
Answer this question but cite a verifiable source (database record, DOI, PMID) for EVERY factual claim. Do not present any claim that you cannot provide a source for; write "must be verified": [question].
3) Made-up sequence scanning:
List all arrays, constants, and variants embedded in the following code:[code]. For each one, clearly mark "must be verified" if it is unclear whether it comes from a real source or is a placeholder you created.
4) Chain control:
Each step builds on the previous one in the following reasoning: [chain].Which subsequent steps collapse if the first link (the underlying data) is wrong? List the KEY data points that the chain needs to verify.
Weak prompt / Strong prompt
Weak: "Tell us everything you know about this variant."
Problem: AI fabricates frequency, attribution, clinical claim from memory; There is no validation hook.
Strong: "Respond on this variant; state which source to verify for each factual claim, mark 'must be verified' if unsure, do not fabricate any frequency or attribution."
Why it's powerful: The field of fabrication is narrowed, each claim is tied to the verification hook.
Common mistakes
- Mistaking fluency for accuracy. The most convincing sentences can be the most dangerous hallucinations.
- Building on a single value without validating it. This is how a chain hallucination is born.
- Not reading constants embedded in the code. AI can embed made-up string/constant into the code.
- Being content with "If you don't know, tell me." This instruction reduces probability and does not provide certainty.
- Spending verification budget in the wrong place. Checking the unimportant facts, not the most critical facts.
Caution: Hallucination rate varies by model version, question, and domain; but it is wrong to say "this model doesn't fit anymore". Verification discipline must be maintained regardless of how good the model is. Responsibility always lies with the person.
Depth: Why RAG and 'driving' don't end hallucination
In recent years, many AI tools have used RAG (Retrieval-Augmented Generation — the method by which the model retrieves relevant documents from a database and uses them before generating an answer) or direct tool invocation (e.g. actually searching PubMed) to “link” its answer to real sources. These approaches reduce the hallucination but do not end it for two reasons. First, the model may summarize the captured document incorrectly or attribute a result to the document that is not in the document; Even if the source is real, the interpretation may be fabricated. Second, the search may return the wrong or irrelevant document and the model will still turn it into an authoritative answer. In other words, even an answer that appears to be "sourced" should not be considered reliable without opening and reading that the source actually supports that claim.
A concrete example: a RAG-based assistant returned an actual ClinVar page for a variant but summarized the page as “pathogenic”; However, on the page, the variant was marked as "conflicting comments". The source was correct, the summary was wrong — and a mistake of clinical weight. Second example: a literature tool pulled an actual article, but the article was about a different gene; The model was fooled by the similarity of the title and attributed the result to the question.
Rule of thumb: if a link is given, open the link; if a quote is given, see if the quote is quoted verbatim in the source. “Sourced AI” facilitates verification, not eliminates it. Your verification reflex remains the same regardless of how “connected” the vehicle is.
5) Welded response inspection template:
For each source link/quote you provide in this answer, show me the exact sentence/passage where I can check if the claim occurs EXACTLY in the source. If the source is factual but your summary deviates from it, mark it.
In summary
- Hallucination is a natural consequence of the LLM's modus operandi; It is not an "accident", but an expected phenomenon that must be caught systematically.
- In genetics, AI can make up genes, sequences, variants, frequencies, attributions, coordinates, statistics and clinical claims.
- Context, resource imperative, and self-control reduce but do not end the hallucination; Only primary source control provides accuracy.
- Develop verification reflexes specific to the output type (sequence→FASTA, citation→DOI); Focus your verification budget on the most critical facts.
Application task
Ask the AI about a genetic topic and personally verify each factual claim (gene, sequence, variant, frequency, attribution) in the output against the reflex list above. Count how many claims are true, how many are false/fabricated, and how many are vague. Summarize the result in a table and note which type of claim is most fabricated.
checklist
- [ ] I applied my verification reflex for each type of output.
- [ ] I have personally checked critical facts from the primary source.
- [ ] I confirmed the basic data point in the chained reasoning.
- [ ] I have read and confirmed the arrays/constants embedded in the code.
- [ ] I did not consider the smooth and confident tone as evidence of accuracy.
- [ ] I focused my verification budget on the highest risk claims.