Gains:
- Being able to distinguish where artificial intelligence saves real time in the QA process and where quality decisions such as 'ready for publication' are left to humans, depending on the task risk level
- Ability to recognize the risk of false passes and implement a verification discipline that tests every AI test by deliberately breaking the code
- Ability to protect test data, personal data and keys, and acquire the habit of performing security testing only within authorization and for defensive purposes.
Consider a release night. Hundreds of tests were run, they all got the green light, the team was relieved and the software went live. The next morning, the customer reported that the payment screen had crashed. The tests were green but he didn't see the error. This is the most insidious nightmare of the quality assurance (QA) profession, that is, the discipline that systematically ensures that the software is of the desired quality: the test that glows green but does not actually confirm anything. When artificial intelligence (AI — software that extracts patterns from historical data and generates text and code) enters this profession, there is both a huge acceleration and a magnification of exactly this nightmare. The initial promise of this module is clear: AI is a testing assistant, blueprint generator and idea multiplier; You are the tester who signs off on the "is this software ready for release" decision.
In this first unit we will focus on discipline, not the tool. You'll learn where AI saves real time in the QA process, where it's dangerous, why the deceptive green so-called "false-pass" is the biggest risk, how to verify each output, and what data you can give to which tool. Without laying this foundation, subsequent units will remain in the air.
Where does AI come in handy in the testing process?
Let's divide the testing jobs into two large clusters. First cluster: repetitive, producible, draft jobs. Drafting a test case from a requirement, listing breakpoints, writing an automation code skeleton for a screen, translating a complex error case into a neat error report, summarizing hundreds of lines of log files, extracting a schema from an API response. In these tasks, AI reduces minutes to seconds and does not get tired.
Second cluster: decisions whose outcome is quality, trust and responsibility. Decisions such as “can this version go live”, “is this bug critical or can it be postponed”, “is this test coverage sufficient”, “does this scenario capture real user risk” etc. require context, product knowledge and responsibility. Here the AI generates options, drafts — but you decide “pass/fail” and “go/no go.”
Let's clarify the distinction in one sentence: AI is strong at "what situations can be tested and how to write code that tests it"; The decision is yours when it comes to the question "Does this software really work and who vouches for it?"
Tip: Before handing over a job to the AI, ask: “What happens if this output is wrong and I don't notice?” If the answer is "I'll lose a few minutes", delegate easily. If the answer is "faulty software goes live", let the AI produce the draft and you make the decision and verification.
False pass: the number one risk of AI in QA
When a test lights up green, it can mean two things: either the software is actually working correctly, or it's not seeing the bug because the test was written incorrectly. The second is called a false pass — the test says "pass" but doesn't actually confirm anything. This risk increases significantly in tests produced with AI, because AI is very successful at writing fluent, smooth-looking but empty tests.
The three most common forms of pseudo-pass are: (1) Testing without assertion — the code runs, contains no asserts, always passes. (2) Self-verifying test — the expected value of the test is calculated from the output of the code under test; That is, whatever the code produces, the test accepts as "correct". (3) Test that verifies the wrong thing — assert exists, but it checks something trivial (e.g. "response is not null"), not the actual business rule.
Caution: A green test panel is not proof of quality; At best it says "the controls we wrote are not broken right now". Don't be comforted by seeing a "pass" on the test the AI produces — the real question is: will this test turn red if I deliberately break the code? If it does not rotate, that test is a decoration.
The golden rule that repeats throughout this module: test every AI test by deliberately breaking the code. If the test is still green, that test is not working. (We will deepen this idea as mutation testing in unit 10.)
Verification discipline: three steps
AI speaks with confidence; That doesn't mean it's true. Develop a three-step reflex to apply to every outcome:
- Tie it to the requirement. Every test case and assert that the AI produces must be based on a real requirement or acceptance criteria (conditions that a job must meet in order to be considered "done"). “Which rule does this scenario confirm?” ask.
- See red. Run the generated test once, breaking the code. If it does not turn red, the test is invalid. This is the non-negotiable step in AI testing.
- Pass it through the context filter. Does the output match what you know is product behavior, architecture, actual user flow? Your domain knowledge is the final filter.
Data privacy and security: what goes where?
The data you work with in the test environment is often sensitive: real customer records, production database copies, API keys, internal system addresses, yet-to-be-announced features. Make a simple classification: Open data (documented, publicly available) can enter any vehicle. Internal data (source code fragments, internal documentation) only to agency-approved tools. Confidential data (real customer data, identity information, vulnerability details, keys) only enters the institution's contracted tools, whose data do not go to model training, preferably masked.
There is an additional limit in the context of security testing: everything learned in this module is for defensive purposes — to authoritatively test the security of your own product. Using AI to infiltrate someone else's system without permission, weaponize real vulnerabilities, or test a system for which you have no authority is both unethical and criminal. No offensive testing will be done without authorization (scope and permission).
Tip: Use synthetic (artificially produced) test data instead of real customer data. Asking the AI to “generate realistic but completely fictional test data” both preserves privacy and diversifies edge cases.
three mini cases
Case 1 — Time saver in the right place. The tester of an Ekomerce team spent 6 hours manually creating a test scenario from the 30-page requirements document for each release. He gave the document (the part that did not contain trade secrets) to YZ and asked for a structured scenario draft; The time was reduced to 90 minutes. He devoted the time saved to verifying by himself adding business-rule edge cases that the AI had missed. AI took away the repetitive work, leaving the judgment to the human.
Case 2 — Fake-passing caught. A developer had the AI write 12 unit tests for a compute function; they were all green. The tester implemented the "see red" step: deliberately changing the addition sign inside the function to multiplication. Only 3 out of 12 tests returned red. The other 9 tests provided no real confirmation; It just said "it didn't throw an error". 9 decorative tests were deleted and 5 real tests were written instead.
Case 3 — Return from privacy breach. An intern pasted an error log containing real customer emails and the last four digits of the card from the production database into a public tool and said "explain this error." The QA leader intervened: this was personal data out of control and a violation of KVKK (Personal Data Protection Law). The same work was done in an institution-approved vehicle, masking personal areas and leaving only a stack trace.
Four copyable templates
1) Job suitability assessment:
Your role: senior QA leader. I'll describe a testing job to you. Tell me (1) whether this work is drafting/analysis work that can be safely delegated to the AI or a quality decision that the human must make, (2) the potential cost of incorrect output, (3) the verification I should do before delegating.Job: [insert job here]
2) Pseudo-pass control:
Check out the test below. Tell me:- What behavior does this test confirm? (one sentence)- How can I break the code under test so that the test turns RED?- Is there a weakness that could cause this test to always pass (missing assert, self-validation, trivial check)?Test: [paste test here]
3) Test data masking control:
The log/data I will give you may contain personal or confidential fields (e-mail, name, card, key, internal address). First, list the fields that need to be masked; I will mask it and send it again. Don't analyze it as it is.
4) Synthetic test data generation:
Generate 20 rows of completely fictional, realistic test data for [the following field structure]. Do not use real person/organization data. Also include edge cases: empty space, too long text, limit values, invalid format.
Weak prompt / Strong prompt
Weak: "Write tests on this code."
Strong: "Calculate this Write unit tests for the discount function. Acceptance criteria for the function: 10% discount over 1000 TL, 20% discount over 5000 TL; negative amount should throw an error. Specify with a comment line which rule you are validating for each test. Test the limit values (999, 1000, 1001, 5000, 0, -1) separately. Use real asserts that will turn red if I break the code; empty or don't write trivial assert."
Powerful prompt; It provides acceptance criteria, limit values, validation expectations, and explicit anti-spoofing instructions. The weak prompt invites the AI to write a decorative test.
Common mistakes
- Trusting green. Thinking that passing the test is proof. The real question is: does it turn red when you break the code?
- Requesting a test without giving any reason. AI produces generic, often useless tests without knowing what needs to be verified.
- Skipping verification. Saying "AI wrote it, it's probably true". The responsibility lies with the person using the output.
- Pasting real/sensitive data into the tool. Working with production data, keys or personal data.
- Unauthorized security testing. Attempting offensive testing without scope and permission.
- Using AI to delegate decision making. Asking the question "Can this version be released?" to the AI and putting the answer in the signature.
In summary
AI is a powerful assistant in the QA process that speeds up repetitive and producible work; But the responsibility for the quality decision lies with the human. The number one risk of AI in this profession is pseudo-pass: green tests that look neat but don't confirm anything. Test every AI test by deliberately breaking the code; If it does not turn red, that test is a decoration. Tie it to the requirement, see the red, pass it through the context filter. Mask confidential data, perform security testing only for authorized and defensive purposes.
Application task
Take 5 AI-generated (or AI-generated) unit tests from your own project. For each: (1) write down in one sentence which behavior it verifies, (2) deliberately break and run the code under test and note how many turn red, (3) mark the ones that don't turn red as "decor tests" and rewrite them with the real assert. Put the result in a table: test name / rule it verified / was it broken when broken / action.
checklist
- [ ] Before handing over the work, I asked the question "what will I lose if it goes wrong?"
- [ ] I tested every AI test by breaking the code; I replaced the one that didn't turn red with the real test.
- [ ] I linked the test cases to the actual requirement/acceptance criteria.
- [ ] I masked sensitive/real data without giving it to the tool; I used synthetic data if possible.
- [ ] I considered security testing only within authority and for defensive purposes.
- [ ] I left the decision of "whether the version will be released" to myself, not to the AI.