Gains:
- Understand what professional skepticism is and the risks of falling into confirmation bias and automation bias when using artificial intelligence
- Ability to test artificial intelligence output with a questioning technique that tries to refute (look for the opposite)
- Being able to understand that the fluency of artificial intelligence does not mean accuracy, that the skeptical mind is the core of control and cannot be transferred.
There is one mental attitude that distinguishes independent auditing from many other professions: professional skepticism. The ISAs define this as the auditor working “with an inquiring mind, noting whether the evidence indicates potential misstatement, and critically evaluating the evidence.” That is, the auditor does not automatically accept any information presented to him—not a management statement, not a document, not even an account that “seems reasonable”—as true. I constantly wonder "Is it really so?" he asks. This attitude is the core of control and is inalienable.
In the age of artificial intelligence, this core faces a brand new test. Because AI is adept at triggering two deep-seated weaknesses of the human mind — automation bias and confirmation bias. In this unit, we will cover how an auditor using AI can maintain or even strengthen his skepticism. The main thesis is that AI's fluency is not accuracy; The skeptical mind becomes even more important in the face of AI.
Two dangerous biases
Automation bias is the tendency for people to trust the output of a machine/software more than their own judgment. "The computer calculated it, it is correct." This danger is magnified when it comes to AI, because AI doesn't just calculate; Speaks fluent, confident, human-like, reasoned sentences. A confident tone is persuasive even if the information is not accurate.
Confirmation bias is the tendency of people to seek information that supports the conclusion they previously believed or hoped for and to ignore conflicting information. This is fatal in auditing: An auditor who thinks "This client must be clean" will be relieved when he asks the AI and gets confirmation, and will stop looking for evidence to the contrary. Because AI tends to lean into the question being asked (if you ask “it's okay, right?” it's more likely to produce an answer that agrees with you), it can foster confirmation bias.
There is also a third: hallucination. We saw it many times in the module — AI can fluently make up a number, item, resource that does not exist. The danger is compounded when three weaknesses are combined: AI makes things up (hallucination), you believe it because it is fluid (automation bias), you don't question it because it tells you what you expect (confirmation bias).
The skeptical mind's three tools against AI
Let's put skepticism into concrete techniques against AI:
- Try to refute, not confirm. An AI might ask its output “is it true?” not "how do I know if this is wrong?" question. Ask yourself: “If this statement were false, what evidence would show it?” and look for that evidence. It is easy to find a single confirming example; Trying to refute and failing gives real assurance.
- Ask neutrally and backwards. Ask the AI "no problem here, right?" Don't ask; this invites an approving response. Instead, “what are the risks I might have missed in this data?” and “what are the three most likely scenarios in which this result could be wrong?” ask. Ask the same question from two opposite sides and compare the answers.
- Download to source and reproduce. For each issue, attribution, and conclusion, “where do I independently verify this?” Ask the question and actually verify. View the output of the AI as an assertion; The evidence is external to the claim.
Tip: Make a habit: Next to every significant output the AI gives, write two things by hand — “(1) what did I verify this with, (2) what evidence would show this if it were false.” These two notes break both automation and confirmation bias.
Judgments that the auditor cannot delegate
Some jobs inherently require skepticism and judgment; these cannot be delegated to AI, only supported:
inalienable jurisdiction
Why
Whether the evidence is sufficient and appropriate
Professional judgment; context dependent
Plausibility of an explanation
Requires consideration of enterprise and intent
Fraud risk assessment
Skepticism and intuition; beyond the data pattern
Materiality and opinion decision
Judiciary responsible to the public
Opinion about the honesty of management
Reading human behavior; AI can't do this
The difference between skepticism and pedantry: a measured attitude
Professional skepticism does not mean disbelieving everything or blaming management in advance; This would be an overreach and would render the control ineffective. Skepticism is neither pure trust (accepting everything you are told) nor blind denial (rejecting everything); In between, it is an attitude of inquiry commensurate with the evidence. The auditor may accept management as honest, but tests this acceptance with evidence; may find an explanation plausible, but confirms plausibility. AI can upset this balance: in one direction, it lulls you into overconfidence with its fluid output; On the other hand, if you tell it to "find fault in everything" it will bombard you with endless false positives. The correct attitude is to always weigh the outcome against the weight of the evidence, while running the AI to produce evidence and disprove the evidence. Skepticism is not a personality trait, but a disciplined way of working; And it is the auditor who establishes, maintains and evidences this discipline, not AI.
three mini cases
Case 1 — The rebuttal technique works. An auditor printed out the AI saying “this stock valuation looks reasonable.” Instead of confirming, he asked backhandedly: “If this valuation is overstated, what evidence would show it?” "items for which after-sales prices fall below cost," YZ said. The auditor searched for these items and found a group that was actually undervalued. The rebuttal question exposed the risk that confirmation concealed.
Case 2 — Falling into confirmation bias. A team member asked the AI during a crunch time, “there's nothing major wrong with this account, right?” he asked. YZ leaned on the question and said, "According to the information you provided, there does not seem to be a significant problem." The team member relaxed and capped the pen. However, he had not presented any evidence; The AI merely mirrored his sentence. The person responsible showed that the question was not asked impartially. Lesson: how you ask the question determines the answer you get; A question seeking confirmation produces confirmation.
Case 3 — Falling for fluency. An auditor thought that an interpretation of legislation that YZ explained in a very fluent and reasoned paragraph was correct and included it in the working paper. The text was so neat that it didn't occur to him to question it. The person responsible showed that the reference had no equivalent in the official text. Lesson: fluency and confidence are not proof of accuracy; The outcome that seems most convincing may be the most dangerous.
Weak prompt / Strong prompt
Weak prompt:
This analysis is correct, right? Do you approve?
The problem: a biased question that seeks confirmation. It pushes the AI to join you; It fosters confirmation bias and hides true risk.
Powerful prompt:
Your role: you are an independent auditor's critical inquiry partner. Your task is not to AGREE with me, but to try to REFUSE my output.Context: Below is my conclusion of an analytical procedure and my tentative conclusion.[result text]Task:1) List the 3 most likely scenarios in which this conclusion could be FALSE.2) For each scenario: "if this were true, what evidence would show it?" Suggest concrete, searchable evidence. 3) Mark any data or assumptions I may have missed. 4) Don't make sentences that agree with me; You are tasked with finding weak points. Mark "unknown" where you are not sure.
This prompt is powerful because it reverses the role of the AI (rebuttal), requiring rebuttal scenarios and searchable evidence, and prohibiting confirmation-seeking language. This puts AI in the service of skepticism rather than an instrument of confirmation bias.
Common mistakes
- Asking questions seeking approval. "It's okay, right?" by asking and ensuring the agreed answer.
- Mistaking fluency for accuracy. Accepting well-written, confident output without question.
- Be content with a single confirmatory example. Closing with the first supporting finding, without trying to refute it.
- To delegate inalienable jurisdiction. Leaving the decision of evidence sufficiency, reasonableness, opinion to AI.
- Letting go of skepticism under time pressure. Taking refuge in the comforting answer of AI when you are stuck.
Caution: Time pressure is the greatest enemy of skepticism, and AI becomes most dangerous under this pressure; because it offers you a fast, fluent and comforting answer. Make it a must to ask the rebuttal question when you're in a rush.
In summary
Professional skepticism is at the core of auditing and becomes even more critical in the age of AI. AI triggers three weaknesses: hallucination (makes it up), automation bias (we believe it because it's fluid), confirmation bias (we don't question it because it says what we hope for). The antidote is to rush to disprove AI, not confirm it: “if this is false, what evidence would show it?” ' and looking for contrary evidence, asking objectively and backwards, reproducing every output from the source. Judgments such as sufficiency of evidence, reasonableness, risk of fraud, and opinion judgment require skepticism and are not transferable. Fluency is not accuracy; The skeptical mind is yours.
Application task
Get your audit result (or a hypothetical result). Ask the AI first with a "confirmation-seeking" weak question, then with the "rebuttal" strong prompt pattern above. Put the two outcomes side by side and write down the difference: which risks were revealed only by the rebuttal question? Then add two notes next to each key claim in your output: "what did I verify with" and "what evidence would show if it were false".
checklist
- [ ] I asked the AI rebuttal/neutral questions, not confirmation-seeking questions.
- [ ] For each significant outcome, “if this is false, what evidence would show it?” I asked the question and looked for evidence.
- [ ] I tested the output, which seemed smooth and confident, with extra skepticism.
- [ ] I did not stop with a single confirmatory example; I tried to refute it but failed.
- [ ] I made my own judgments such as sufficiency of evidence, reasonableness, and opinion.
- [ ] Even under time pressure, I did not skip the rebuttal question.
- [ ] Next to the important claims, I made notes of "what I verified with / what would show if it was wrong".