Gains:
- Ability to have artificial intelligence produce multiple choice questions, distractors based on typical errors, and solution keys by specifying topic, level, and difficulty distribution.
- Ability to catch wrong key and multiple/zero correct option errors by independently validating each question and answer key, preferably in bulk, with SymPy
- Ability to evaluate the quality of distractors, the unambiguity and difficulty balance of questions, and approve each question before taking the exam.
Preparing exam and practice questions is one of the most time-consuming tasks in mathematics education: setting questions of appropriate difficulty, designing plausible distractors (wrong but plausible options), preparing the answer key, and balancing difficulty take hours. Artificial intelligence can speed up this process — generating dozens of question variations, multiple choice options, and solution keys from one topic. But the accuracy of each question produced and the error-freeness of each answer key must be independently verified; An incorrect key means students unfairly lose points. In this unit you will learn how to use AI as a question generation partner and how to moderate each question.
A few definitions. A distractor is an option in a multiple choice question that is not correct but corresponds to a common mistake the student may make. A good distractor is not random; Reflects a typical error (sign error, rule confusion). The answer key is a list of correct answers to the questions. Difficulty balance is the appropriate distribution of easy-medium-difficult questions in an exam.
Contribution of AI to question generation
- Question variation: Multiple similar questions from a pattern (changing the numbers).
- Distractor design: Plausibly incorrect choices based on common errors.
- Solution key outline: Step-by-step solution to each problem.
- Difficulty rating: Classifying questions as easy/medium/difficult.
- Subject coverage: Sets of questions covering different sub-topics of a syllabus.
- Bloom taxonomy fit: Question according to cognitive levels such as remembering, applying, analyzing.
Step by step: generating a reliable set of questions
1. Specify the topic, level and number of questions. Like "10th grade high school, quadratic equations, 10 multiple choice questions."
2. Ask for question type and difficulty distribution. Ask for a distribution like "4 easy, 4 medium, 2 hard."
3. Solve each question independently. Solve each question yourself or with SymPy; Compare it with the answer key provided by YZ. This step is non-negotiable.
4. Check for distractors. Are wrong options really wrong? Sometimes the AI produces more than one correct option, or a distractor is actually correct. Check each option.
5. Guarantee there is exactly one correct answer. In multiple choice, exactly one option must be correct; There must be no zero or more than one correct.
6. Check for uncertainty and duplication. Is the question univocal? Do the two questions measure the same thing? Are the numbers reasonable (e.g. if the root should be negative)?
Attention: AI makes two most common mistakes when generating multiple choice questions: (1) the option it marked "true" is actually wrong, (2) more than one option is correct, or none of them is correct. Do not use any question without solving each question independently and evaluating each of the four options separately as true/false.
Question quality control chart
control
Question
what to search for
accuracy
Is the answer key correct?
Confirmation by independent solution
only true
Is only 1 option correct?
The other 3 must be wrong
Distractor quality
Are the mistakes plausible?
Should be based on typical error
Clarity
Is the question univocal?
There should be no uncertainty
Level compliance
Is the difficulty appropriate to the goal?
Neither too easy nor too difficult
Scope
Is the topic well represented?
Subtopics are balanced
three mini cases
Case 1 — Wrong key. A teacher produced a multiple choice set of 12 questions. When he solved each question with SymPy, he found that in 3 of the 12 questions, the option the AI had marked "correct" was incorrect — the actual correct answer was another option. The teacher straightened the keys. If this set had been used without verification, the entire class would have been evaluated unfairly in 3 questions.
Case 2 — Two correct options. In one question, the AI put in both “x = 2” and “x = 2 or x = −2”; Both were true in a sense, and the question was unclear. The teacher rearranged the options and left only one correct answer. Distractors must be clearly incorrect.
Case 3 — Weak distractor. In an engineering class, the AI placed irrelevant distractors such as "42", "aksaray", "blue", "17" etc. into a question; The correct answer was obviously "17". The teacher asked for numerical distractors based on typical calculation errors (e.g., value resulting from sign error), and the questions became true discriminators. The good distractor represents a true false path.
Four copyable templates
1) Multiple choice question set:
Generate [n] MULTIPLE CHOICE questions on [topic]. Level:[level]. Difficulty: [x easy, y medium, z hard]. Let each question have 4 options; Let EXACT 1 be correct and the other 3 are plausible distractors based on typical errors. Also write step by step solution and correct answer for each question. I will verify all of them with SymPy.
2) Question variation (from pattern):
Take this question as PATTERN: [sample question]. Generate [n] different versions measuring the same concept but with varying numbers/context. Solve the answers for each version step by step. Let the answers be different from each other.
3) Solution key generation:
Produce a STEP-BY-STEP solution and final answer for each of the questions below. Specify the rule you use. Mark the solution you are unsure of; I will independently verify each answer. Questions: [here]
4) Distractor improvement:
Review the distractors for this multiple choice question. Explain WHICH typical student error each incorrect distractor corresponds to. Replace irrelevant or obvious incorrect distractors with ones based on actual errors. Question: [here]
Weak prompt / Strong prompt
Weak: "Do me 10 derivative questions."
Result: Uncertain level, random difficulty distribution, unverified answer key, random set of distractors (if any).
Strong: "Produce 10 multiple choice derivative questions for the 12th grade of high school: 4 easy (basic rules), 4 medium (chain/product rule), 2 difficult (compound functions). Let there be 4 choices in each question, exactly 1 correct, the others are distractors based on typical errors (forgetting the inner derivative, sign error). Solve each question step by step and indicate the correct answer."
Result: A set that is level, balanced, distractors are meaningful and every answer is auditable.
Common mistakes
- Not verifying the answer key. What AI says is "right" is often wrong; Solve each question independently.
- More than one/zero is the correct option. In multiple choice there must be exactly one truth; Check each option separately.
- Weak distractors. Irrelevant or obviously incorrect options make the question undistinctive.
- Not specifying the difficulty distribution. They all appear to be the same difficulty; Level evaluation cannot be made.
- Repetition and overlap. Different questions may measure the same concept; Check coverage balance.
- Vague expression. Questions with double meanings unfairly mislead students.
Tip: The most efficient way to validate a set of questions is to write code that solves all the questions at once with SymPy. Tell the AI "write code that solves each of these 10 problems with SymPy", run the code, and bulk compare the SymPy results with the AI's answer key. This way, you check the 10 questions not one by one, but collectively and deterministically.
In summary
AI dramatically speeds up math question and exam generation: question variation, distractor design, solution key, difficulty rating. But each question and each answer key must be independently validated — preferably bulk-verified with SymPy. The most common errors are wrong answer key, multiple/zero correct options, and weak distractors. Specifying the topic, level and difficulty distribution from the beginning increases the quality. Any unverified questions should not enter the exam.
Application task
Choose a topic. Have the AI produce 8-10 multiple choice questions + solutions with a certain difficulty distribution with the 1st template. Then, remembering from unit 4, have the AI write a code that solves all the questions with SymPy and run the code. Compare SymPy results with YZ's answer key. Verify that there is exactly one correct choice in each question and that the distractors are actually incorrect. Find and correct at least one error or weak distractor.
checklist
- [ ] I have clearly stated the subject, level and difficulty distribution.
- [ ] I solved each question independently (preferably batch with SymPy).
- [ ] I compared the answer key with the SymPy results.
- [ ] I verified that there is exactly one correct option in each question.
- [ ] I checked that the distractors were based on typical errors and were actually wrong.
- [ ] I made sure there were no vague statements and no repetitive questions.