Skip to content

Lesson design and AI authoring

Part of the AI coach reading library: 32 papers cited in the AI coach notes, each in plain language. Who was studied, how, what it found, why it matters for our students, and where our notes use it. Active learning and Peer Instruction in physics, struggle-first teaching, and how well AI writes solutions, hints and questions. Download links open the archived copy and need an Avanti Google login; the Source link is the public original.

The overviews are drawn from the extraction tables and literature reviews written when each paper was read, not from a fresh reading. Where a number here and in a note differ, the note’s section is the one to check.

Active learning and Peer Instruction in physics

Section titled “Active learning and Peer Instruction in physics”

Interactive-Engagement versus Traditional Methods. Do physics courses where students answer and discuss teach concepts better than lectures?

Section titled “Interactive-Engagement versus Traditional Methods. Do physics courses where students answer and discuss teach concepts better than lectures?”

Hake 1998, American Journal of Physics. Download · Source

  • Who was studied: 6,542 students in 62 introductory mechanics courses at US high schools, colleges and universities, including 14 high-school courses with 1,113 students.
  • How: Survey of before-and-after scores on the Force Concept Inventory (FCI, a multiple-choice test of basic ideas about force and motion) and its predecessor, sent in by instructors. Results are given as the normalised gain: the share of the possible improvement a class actually made.
  • What it found: 14 traditional courses averaged a normalised gain of 0.23 (SD across courses 0.04). 48 interactive-engagement courses averaged 0.48 (SD 0.14). The 14 high-school interactive courses averaged 0.55 (SD 0.11). The data were volunteered, so they lean towards good courses; Hake says so himself.
  • Why it matters for us: Lessons where the student must answer, predict or explain teach more physics concepts than lessons where the student only listens, in schools as well as universities.
  • Where we use it: Lesson design literature review: evidence table, and design rule 7 (mix “why” and “how” questions, because courses that raised conceptual scores also raised problem-solving scores).

Peer Instruction: Ten Years of Experience and Results. Does voting and discussing concept questions in lectures raise physics learning?

Section titled “Peer Instruction: Ten Years of Experience and Results. Does voting and discussing concept questions in lectures raise physics learning?”

Crouch and Mazur 2001, American Journal of Physics. Download · Source

  • Who was studied: Harvard introductory physics students who were not physics majors, 117 to 246 a year over ten years; 298 students in the 1990-91 comparison.
  • How: Year-on-year comparison, not randomised. Peer Instruction gives a concept question, one to two minutes to answer alone, two to four minutes of discussion with a neighbour, then a revote and an explanation. Later years added other changes.
  • What it found: In the calculus-based course the normalised FCI gain rose from 0.25 (traditional, 1990) to 0.49-0.74 (Peer Instruction, 1991-1997). In the algebra-based course it rose from 0.40 to 0.65 and 0.63. Scores on the problem-solving Mechanics Baseline test rose from 66% to 71-79%. Discussion helped most when 35% to 70% of students answered right at first, with the largest gain near 50%.
  • Why it matters for us: Spending lesson time on concept questions did not cost problem-solving marks, but the gain comes from a package that includes peer discussion, which a one-to-one coach cannot offer.
  • Where we use it: Lesson design literature review, design rules 1, 7, 8 and 12. Meta-analysis section 3.8 (“the commit step has no evidence of its own”). What works, section a) (nearly half of correct answers came only after discussion).

Improved Learning in a Large-Enrollment Physics Class. Does one week of active teaching beat a lecture on the same topic?

Section titled “Improved Learning in a Large-Enrollment Physics Class. Does one week of active teaching beat a lecture on the same topic?”

Deslauriers, Schelew and Wieman 2011, Science. Download · Source

  • Who was studied: Two sections of first-year calculus-based physics for engineers at the University of British Columbia, Canada, 267 and 271 students, during one week on electromagnetic waves.
  • How: Quasi-experiment (two existing sections compared, not randomised). One section had pre-class reading, clicker concept questions, group tasks and immediate instructor feedback; the other had lectures.
  • What it found: End-of-week test 74% against 41% (guessing would give 23%), an effect of 2.5 SD (standard deviations, the usual unit for comparing test-score gains). Attendance was 75% against 53% and engagement 85% against 45%. The test was written by the first author, the instructors differed and the comparison lasted one week.
  • Why it matters for us: It supports the format, but 2.5 SD is not a size to plan around for our students.
  • Where we use it: Lesson design literature review: evidence table and the “Commit, then compare, then hear the answer” section.

Measuring Actual Learning versus Feeling of Learning. Do students learn more from an active class, and do they know it?

Section titled “Measuring Actual Learning versus Feeling of Learning. Do students learn more from an active class, and do they know it?”

Deslauriers, McCarty, Miller, Callaghan and Kestin 2019, PNAS. Download · Source

  • Who was studied: 149 Harvard introductory physics students, over two class meetings (statics and fluids).
  • How: Randomised crossover (each student had both kinds of class, in random order). In the active class students first worked the problems in small groups, then saw the same full solution that the lecture class saw.
  • What it found: The active class scored 0.46 SD higher on a test of learning and rated its feeling of learning 0.56 SD lower. A 20-minute talk early in the semester explaining this led over 65% of students to report that their view of active learning improved.
  • Why it matters for us: A student’s sense of having understood a lesson, or their score during it, cannot be used as a sign of mastery.
  • Where we use it: Lesson design literature review, design rules 6 and 11. Meta-analysis section 3.10 (students rate active lessons lower even when they learn more) and section 4.1, implication 5. Coach walkthrough, “What to look for afterwards”.

Active Learning Increases Student Performance in Science, Engineering, and Mathematics. Across many studies, how much does active learning beat lecturing?

Section titled “Active Learning Increases Student Performance in Science, Engineering, and Mathematics. Across many studies, how much does active learning beat lecturing?”

Freeman, Eddy, McDonough, Smith, Okoroafor, Jordt and Wenderoth 2014, PNAS. Download · Source

  • Who was studied: Undergraduate STEM courses, mostly in the US: 225 studies, 158 on scores and 67 on failure rates.
  • How: Meta-analysis (a statistical pooling of many studies’ results) of active learning against traditional lecturing.
  • What it found: Exam and concept-inventory scores rose by 0.47 SD. The failure rate was 21.8% under active learning against 33.8% under lecturing (odds ratio of failing under lecture 1.95). Effects were larger on concept inventories than on instructor-written exams and largest in classes of 50 or fewer. “Active learning” covers very different activities.
  • Why it matters for us: Doing beats listening in general, but this is undergraduate evidence and does not say which order of activities is best.
  • Where we use it: Lesson design literature review: evidence table and “Interactive engagement beats listening”. Meta-analysis section 3.13 (what happens in the lesson matters more than the medium).

Oersted Medal Lecture 2001: Physics Education Research, the Key to Student Learning. How should physics tutorials be built from students’ errors?

Section titled “Oersted Medal Lecture 2001: Physics Education Research, the Key to Student Learning. How should physics tutorials be built from students’ errors?”

McDermott 2001, American Journal of Physics (read in the University of Washington Physics Education Group’s selected-readings packet). Download · Source

  • Who was studied: US undergraduates using Tutorials in Introductory Physics, with graduate teaching assistants as a comparison group. On the example shown: about 560 students before, 405 after, and 55 teaching assistants.
  • How: A synthesis of a research programme, with before-and-after data and no control group. Each tutorial uses a pretest to bring out the error, then a group worksheet that leads students to notice an inconsistency and reason through it in steps “of just the right size”.
  • What it found: On one interference item, correct answers with correct reasoning rose from under 5% before the tutorial to 40% after, against 25% for graduate teaching assistants before any tutorial. Warning students not to make a particular error is ineffective, and a single encounter is seldom enough.
  • Why it matters for us: Telling a student “don’t make this mistake” will not fix it; the lesson has to bring out the error and walk the student through it in small steps, and come back to it later.
  • Where we use it: Lesson design literature review, design rules 1, 4, 6, 9, 10, 12 and 13. Meta-analysis section 3.8 (telling students “don’t make this mistake” does not work).

Investigative Science Learning Environment. Does an observe-predict-test cycle work as a physics course design?

Section titled “Investigative Science Learning Environment. Does an observe-predict-test cycle work as a physics course design?”

Etkina and Van Heuvelen 2007, PER Reviews vol. 1 (book chapter). Download · Source

  • Who was studied: Several US university physics courses using ISLE.
  • How: A description of ISLE, in which students observe, propose explanations, predict the outcome of a testing experiment, then apply the idea, with the developers’ own outcome reports. No controlled comparison.
  • What it found: FCI normalised gains of 0.60 and 0.64 in two ISLE courses; 63-74% on an electricity concept survey against a 47% average for calculus-based courses; 73% against 58% on eight reused final-exam MCQs.
  • Why it matters for us: A predict-then-test cycle is a usable skeleton for a lesson, though the evidence is the developers’ own.
  • Where we use it: Lesson design literature review: evidence table and “Elicit, confront, resolve”.

Force Concept Inventory. How do you build a test whose wrong answers reveal students’ beliefs?

Section titled “Force Concept Inventory. How do you build a test whose wrong answers reveal students’ beliefs?”

Hestenes, Wells and Swackhamer 1992, The Physics Teacher. Download · Source

  • Who was studied: Over 1,500 US high-school students and over 500 university students.
  • How: Development and administration of the FCI. Each wrong option is written to match a common-sense belief about force and motion.
  • What it found: The authors propose 80% as the threshold for Newtonian thinking and report a threshold near 60% below which students could not use Newtonian ideas to solve problems.
  • Why it matters for us: Every wrong option in a coach question should stand for a named belief, and a conceptual floor is needed before procedure practice pays off.
  • Where we use it: Lesson design literature review, design rules 2 and 9. Listed in the meta-analysis references.

A Time for Telling. Does analysing contrasting cases prepare students to learn from a lecture?

Section titled “A Time for Telling. Does analysing contrasting cases prepare students to learn from a lecture?”

Schwartz and Bransford 1998, Cognition and Instruction. Download · Source

  • Who was studied: US college students with no prior psychology, in three classroom experiments of 24, 18 and 36 students (about 12 per arm in Experiment 3).
  • How: Students were randomly assigned to analyse contrasting cases (examples that differ in the feature that matters) then hear a lecture, analyse the cases twice with no lecture, or summarise a text then hear the lecture. The outcome was a prediction task about a week later.
  • What it found: Experiment 3: cases then lecture gave 43.8% of possible predictions; cases twice with no lecture 16.7%; summary then lecture 14.6%.
  • Why it matters for us: Comparing two cases helps a student learn from the explanation that follows, but neither the comparison nor the explanation works well alone.
  • Where we use it: Lesson design literature review, design rules 3 and 4. Meta-analysis section 3.8 (contrasting cases have promise, on thin evidence).

Productive Failure. Does trying a problem before being taught help, and when?

Section titled “Productive Failure. Does trying a problem before being taught help, and when?”

Kapur and Roll 2023, book chapter in “In Their Own Words” (APA Division 2). Download · Source

  • Who was studied: The chapter reports Kapur 2012, 133 Grade 9 students in Singapore learning standard deviation over four 50-minute periods, and summarises the Sinha and Kapur 2021 meta-analysis (over 12,000 participants in 166 comparisons).
  • How: Kapur 2012 was a quasi-experiment: students spent 40 minutes trying to invent a measure of consistency before being taught, against a direct-instruction group. The meta-analysis figures are its author’s summary.
  • What it found: No group invented the correct formula, but productive-failure students did better on conceptual and transfer items and the same on procedural items. As reported for the meta-analysis: d = 0.36 [0.20, 0.51] (d is an effect size in standard deviations) on conceptual knowledge and transfer, up to 0.58 with high fidelity to the design, with procedural knowledge not compromised. A task takes two to three iterations to get right.
  • Why it matters for us: Struggle first helps only when followed by an explanation that compares the student’s attempt with the correct method, and the task itself needs careful design.
  • Where we use it: Lesson design literature review, design rules 3, 4 and 13. The meta-analysis cites Sinha and Kapur 2021 directly, not this chapter.

Why Minimal Guidance during Instruction Does Not Work. Should novices be left to discover solutions?

Section titled “Why Minimal Guidance during Instruction Does Not Work. Should novices be left to discover solutions?”

Kirschner, Sweller and Clark 2006, Educational Psychologist. Download · Source

  • Who was studied: No new study; a narrative review.
  • How: Argument from cognitive load theory (the idea that working memory can hold only a few new elements at once).
  • What it found: No pooled effect. Argues that minimal guidance is less effective for novices, that worked examples beat unguided problem solving until learners become expert, and that guidance should fade as expertise grows.
  • Why it matters for us: A weak student on a long JEE procedure should see a worked step before being asked for one.
  • Where we use it: Lesson design literature review, design rule 5. Listed in the meta-analysis references.

When Instructional Guidance Is Needed. Does showing a worked example beat letting students generate the answer, and does task complexity matter?

Section titled “When Instructional Guidance Is Needed. Does showing a worked example beat letting students generate the answer, and does task complexity matter?”

Chen, Kalyuga and Sweller 2016, Educational and Developmental Psychologist (accepted manuscript). Download · Source

  • Who was studied: 50 Year 6 students (about 12 years old) learning geometry in Chengdu, China.
  • How: Randomised experiment crossing the level of guidance (worked example or generate the solution) with element interactivity (how many parts of a task must be handled at once).
  • What it found: On the low-interactivity task, generating beat being shown, d = 0.57. On the high-interactivity task, the worked example scored higher, d = 0.46, p = .06 one-tailed (not significant on its own). The interaction between guidance and task complexity was significant (p = .028).
  • Why it matters for us: For a multi-step JEE procedure, show a worked step first; for a single conceptual question, let the student try first. Small sample and young students.
  • Where we use it: Lesson design literature review, design rule 5. Listed in the meta-analysis references and recorded in its effect-size table (meta/effects.tsv).

Developing Conceptual Understanding and Procedural Skill in Mathematics: An Iterative Process. Does understanding come before skill, or do they build each other?

Section titled “Developing Conceptual Understanding and Procedural Skill in Mathematics: An Iterative Process. Does understanding come before skill, or do they build each other?”

Rittle-Johnson, Siegler and Alibali 2001, Journal of Educational Psychology. Download · Source

  • Who was studied: US Grade 5 and 6 students learning decimal fractions on number lines, in two experiments of 74 and 117.
  • How: Experiments tracking conceptual and procedural knowledge over time; Experiment 2 made a correct problem representation easier to form.
  • What it found: Initial conceptual knowledge predicted gains in procedure, and procedural gains predicted later conceptual gains. Supporting a correct representation improved procedural knowledge. Neither kind of knowledge was complete at the start or the end.
  • Why it matters for us: A lesson should move between “why” and “how” questions rather than finish one before starting the other, and mastery should test both.
  • Where we use it: Lesson design literature review, design rule 7.

How well AI solves problems and writes solutions, hints and lesson plans

Section titled “How well AI solves problems and writes solutions, hints and lesson plans”

Have LLMs Advanced Enough? (JEEBench). Can large language models solve JEE Advanced problems?

Section titled “Have LLMs Advanced Enough? (JEEBench). Can large language models solve JEE Advanced problems?”

Arora, Singh and Mausam 2023, arXiv preprint. Download · Source

  • Who was studied: No students. GPT-4 (2023) and older models on 515 past JEE Advanced physics, chemistry and maths problems.
  • How: Benchmark. Models were run with chain of thought (writing out reasoning steps) and self-consistency (taking the majority answer across several samples), and asked to critique their own answers.
  • What it found: Best score 38.9% (GPT-4, chain of thought, 8-sample self-consistency). In 100 checked solutions: 34 conceptual, 30 computation and 15 grounding errors. Self-critique lowered the score from 35.0% to 33.9%; the verifier fixed 2 of 80 wrong solutions.
  • Why it matters for us: Asking a model to check its own JEE solution is not a check.
  • Where we use it: AI authoring literature review, design rules 2 and 3, and the “not yet possible” list (a model checking its own work).

mmJEE-Eval: A Bilingual Multimodal Benchmark. How well do current AI models solve JEE Advanced, in English and Hindi?

Section titled “mmJEE-Eval: A Bilingual Multimodal Benchmark. How well do current AI models solve JEE Advanced, in English and Hindi?”

Mukherjee and Ghosh 2025, arXiv preprint. Download · Source

  • Who was studied: No students. 17 vision-language models (July-October 2025 versions) on 1,460 JEE Advanced questions from 2019-2025, in English and Hindi, with official marking.
  • How: Benchmark, with a held-out 2025 paper and experiments on whether models detect and fix their own errors.
  • What it found: On the held-out 2025 set, first-try accuracy was 72.7% for o3, 79.5% for GPT-5 and 79.8% for Gemini 2.5 Flash. GPT-5 scored 291 of 360 marks against the topper’s 332. Models detected their own errors in 52.9% of cases and corrected 8% of those. GPT-5 was right in English but wrong in Hindi on 7.3% of items. Claude was not tested.
  • Why it matters for us: About one JEE Advanced answer in five is wrong at the first try, so every generated solution must be checked against a verified key, and Hindi adds errors.
  • Where we use it: AI authoring literature review, design rules 2, 3, 9 and 10, and the light-review and heavy-review lists.

Could an Artificial-Intelligence Agent Pass an Introductory Physics Course? How did early ChatGPT do as a physics student?

Section titled “Could an Artificial-Intelligence Agent Pass an Introductory Physics Course? How did early ChatGPT do as a physics student?”

Kortemeyer 2023, arXiv preprint. Download · Source

  • Who was studied: No students. ChatGPT (January 2023) working through a calculus-based introductory physics course at ETH Zurich.
  • How: Case study: the model’s FCI answers, 76 homework problems and exams were graded like a student’s.
  • What it found: FCI 60%; homework 55% (1.88 attempts); exams 47%; course grade 1.5, a narrow pass. Typical errors were plug-and-chug and arithmetic.
  • Why it matters for us: A historical baseline that shows the kinds of error a reviewer should look for.
  • Where we use it: AI authoring literature review, “Can current models solve JEE problems reliably?”

Educational Data Augmentation in Physics Education Research Using ChatGPT. Can ChatGPT answer a physics test like a student would?

Section titled “Educational Data Augmentation in Physics Education Research Using ChatGPT. Can ChatGPT answer a physics test like a student would?”

Kieser, Wulff, Kuhn and Küchemann 2023, Physical Review Physics Education Research. Download · Source

  • Who was studied: No students. ChatGPT on GPT-4, in repeated runs of the 30 FCI items.
  • How: Lab study: the model answered as itself and as simulated students, either of a given cohort or holding a named preconception.
  • What it found: It averaged 83% on the FCI. Prompting it as a student of a given cohort produced no variation; prompting it to hold a named preconception produced varied answers that resembled real students in some respects.
  • Why it matters for us: A model can role-play a named misconception, which helps when piloting wrong options, but it cannot play “a weaker student” on request.
  • Where we use it: AI authoring literature review, “Can current models solve JEE problems reliably?” and “Difficulty”.

Improving the Validity of Automatically Generated Feedback via Reinforcement Learning. Can an AI judge tell whether feedback on a wrong answer is correct?

Section titled “Improving the Validity of Automatically Generated Feedback via Reinforcement Learning. Can an AI judge tell whether feedback on a wrong answer is correct?”

Scarlatos, Smith, Woodhead and Lan 2024, AIED 2024 (conference paper). Download · Source

  • Who was studied: 1,956 Eedi middle-school maths MCQs with teacher-written feedback for each wrong option; 80 messages hand-checked.
  • How: Method study. GPT-4 was used as a rubric judge of feedback and compared with human labels.
  • What it found: The judge matched human labels on “feedback is correct” with accuracy 0.77 (F1 0.81; F1 is a 0-1 score combining how many errors are caught and how many flags are right). Two humans agreed at kappa 0.53 (agreement beyond chance, 0-1). Teacher-written feedback was itself rated correct 91% of the time. The judge and the feedback writer were both GPT-4, which the authors warn may inflate scores.
  • Why it matters for us: An AI judge catches most but not all wrong feedback, and even human-written feedback needs checking.
  • Where we use it: AI authoring literature review, “Hints, worked solutions and step feedback” and the weak-or-contested list (AI judges grading AI output).

Are Large Language Models for Education Reliable for All Languages? Do AI models lose accuracy on teaching tasks in Hindi?

Section titled “Are Large Language Models for Education Reliable for All Languages? Do AI models lose accuracy on teaching tasks in Hindi?”

Gupta, Pal Chowdhury, Zouhar, Rooein and Sachan 2025, arXiv preprint. Download · Source

  • Who was studied: No students. Six models, including Claude 3.7 Sonnet and GPT-4o, on four Eedi-based teaching tasks in English and eight other languages including Hindi and Telugu, machine-translated.
  • How: Benchmark comparing accuracy across languages and with prompts in English or the target language.
  • What it found: GPT-4o, misconception identification: 97.6% in English against 94.5% in Hindi. Feedback selection: 53.4% against 48.7%, and 32.1% when the prompt was also in Hindi. Average across tasks: English 70.9%, Hindi 63.1%. Part of the drop may come from machine translation of the items.
  • Why it matters for us: Keep authoring prompts in English and have a teacher read any Hindi or Marathi output.
  • Where we use it: AI authoring literature review, design rule 10 and the heavy-review list (student-facing Hindi or Marathi content).

Lesson Planning Using AI Lesson Assistant, Aila: A Teacher Choices Trial. Does an AI lesson assistant save teachers time without lowering quality?

Section titled “Lesson Planning Using AI Lesson Assistant, Aila: A Teacher Choices Trial. Does an AI lesson assistant save teachers time without lowering quality?”

Roy, Poet, Staunton, Aston, Thomas and Styles 2026, EEF / NFER evaluation report. Download · Source

  • Who was studied: 464 Years 3-6 (primary) teachers in 108 schools in England, over 10 weeks in autumn 2025.
  • How: Cluster RCT (randomised controlled trial in which whole schools were assigned by lottery) of Aila, Oak National Academy’s lesson assistant grounded in its own curriculum.
  • What it found: Preparation fell from 3.3 to 2.5 hours a week (48.6 minutes saved; ratio 76%, range 64% to 89%), with resource quality no worse. Teachers used it for 7% of lessons, and fewer than 1 in 5 said it changed their teaching.
  • Why it matters for us: AI help grounded in a quality-checked bank can save a teacher time, but the savings came with a teacher adapting the output, not with AI lessons sent straight to students.
  • Where we use it: AI authoring literature review, design rule 12 and the light-review list (teacher-facing lesson outlines).

Auto-Evaluation: A Critical Measure in Driving Improvements in Quality and Safety of AI-Generated Lesson Resources. Can an AI judge rate AI-written lessons as teachers would?

Section titled “Auto-Evaluation: A Critical Measure in Driving Improvements in Quality and Safety of AI-Generated Lesson Resources. Can an AI judge rate AI-written lessons as teachers would?”

Clark, Dowland, Benton, Budai, Keskin, Searle, Gregory, Hodierne, Gayne and Roberts 2024, MIT Open Learning AI + Open Education (also arXiv:2502.10410). Written by the tool’s developer, Oak. Download · Source

  • Who was studied: 4,985 Aila lessons; 311 quiz MCQs rated by 19 Oak teachers.
  • How: Method study: an AI judge (GPT-4o) scored lessons on 24 benchmarks and was compared with teachers on quiz distractor quality (distractors are the wrong options in a multiple-choice question).
  • What it found: Before tuning, the judge scored lower than teachers on 75% of quizzes, with QWK 0.17 (QWK is an agreement score where 1 is perfect and 0 is chance). After adding examples drawn from teachers, QWK rose to 0.32 and the judge scored lower on 62%.
  • Why it matters for us: Any AI judge of question quality must first be calibrated against our teachers, and even then it agrees only moderately.
  • Where we use it: AI authoring literature review, design rule 4 and “Lesson plans and teacher time”.

AI-written questions, wrong options and difficulty

Section titled “AI-written questions, wrong options and difficulty”

Exploring Automated Distractor Generation for Math Multiple-Choice Questions via Large Language Models. Can AI write the wrong options students actually pick?

Section titled “Exploring Automated Distractor Generation for Math Multiple-Choice Questions via Large Language Models. Can AI write the wrong options students actually pick?”

Feng, Lee, McNichols, Scarlatos, Smith, Woodhead, Otero Ornelas and Lan 2024, arXiv preprint. Download · Source

  • Who was studied: About 1.4K Eedi maths MCQs on the “Number” topic; 20 rated by two graduate students.
  • How: Method study comparing several models and prompts for writing distractors.
  • What it found: The best method, which retrieves similar items as examples, matched all three teacher distractors on 11% of items and at least one on 74%. Raters scored AI distractors lower than teachers’: validity 3.28 against 3.99 and plausibility 2.68 against 3.72 (1-5 scale). Rater agreement was QWK 0.23 to 0.54.
  • Why it matters for us: AI writes wrong answers that are possible, not the ones students choose; retrieving our own similar items is the best cheap fix.
  • Where we use it: AI authoring literature review, design rule 5 and the “not yet possible without Avanti’s own student data” list.

DiVERT: Distractor Generation with Variational Errors Represented as Text. Can a small model trained on teachers’ error labels name misconceptions as well as teachers?

Section titled “DiVERT: Distractor Generation with Variational Errors Represented as Text. Can a small model trained on teachers’ error labels name misconceptions as well as teachers?”

Fernandez, Scarlatos, Feng, Woodhead and Lan 2024, arXiv preprint. Download · Source

  • Who was studied: 1,434 Eedi maths MCQs; 180 error descriptions rated by two teachers.
  • How: Method study: a small open model (7B parameters) fine-tuned on teacher error labels, compared with GPT-4o and with humans.
  • What it found: Teacher-rated error quality: human 3.23, fine-tuned 7B model 3.07 (not significantly different), GPT-4o 2.56 (significantly worse). Rater agreement was low (r = 0.33).
  • Why it matters for us: Training a small model on labelled student errors beats a frontier model at naming misconceptions, and that needs Avanti’s labelled wrong-answer data.
  • Where we use it: AI authoring literature review, design rules 5 and 11 and the “not yet possible” list.

Distractor Generation for Multiple-Choice Questions with Predictive Prompting and Large Language Models. What share of AI-written wrong options can teachers use?

Section titled “Distractor Generation for Multiple-Choice Questions with Predictive Prompting and Large Language Models. What share of AI-written wrong options can teachers use?”

Bitew, Deleu, Develder and Demeester 2023, arXiv preprint. Download · Source

  • Who was studied: 300 secondary-school MCQs from the WeZooz test set (language learning and factual items in sciences, geography, history and biology), rated by 10 teachers.
  • How: Method study: ChatGPT prompted with retrieved similar items wrote distractors, which teachers rated.
  • What it found: 53% of ChatGPT distractors were rated usable as is.
  • Why it matters for us: About half of AI-written wrong options are usable in factual subjects; JEE problem solving is likely harder.
  • Where we use it: AI authoring literature review, the heavy-review list (new questions of any kind).

A Comparative Study of AI-Generated (GPT-4) and Human-Crafted MCQs in Programming Education. How do GPT-4’s multiple-choice questions compare with human ones?

Section titled “A Comparative Study of AI-Generated (GPT-4) and Human-Crafted MCQs in Programming Education. How do GPT-4’s multiple-choice questions compare with human ones?”

Doughty, Wan, Bompelli et al. 2024, ACE 2024 (conference paper). Download · Source

  • Who was studied: No students. 651 GPT-4 and 449 human MCQs for six university Python courses, covering 246 learning objectives.
  • How: Comparative study with rubric annotation of each item.
  • What it found: More than one correct answer: 4.9% of generated items against 1.1% of human ones. A distractor that gave the answer away: 4.0% against 0.9%. Generated items were better aligned to learning objectives. Errors clustered in higher-order (apply, create) items.
  • Why it matters for us: Every AI-written question needs a check for a single correct answer and for options that give it away.
  • Where we use it: AI authoring literature review, design rule 6.

ChatGPT’s Quality: Reliability and Validity of Concept Inventory Items. How many AI-written physics concept questions survive expert review?

Section titled “ChatGPT’s Quality: Reliability and Validity of Concept Inventory Items. How many AI-written physics concept questions survive expert review?”

Küchemann, Rau, Schmidt and Kuhn 2024, Frontiers in Psychology. Download · Source

  • Who was studied: 100 ChatGPT 3.5 concept items on kinematics and forces, and 172 first-year university students who took the kept items alongside the FCI.
  • How: Item development study: first screen, expert rating, then a test with students.
  • What it found: 70 of 100 items were removed at the first screen (unrealistic, duplicate, or with no or two correct answers); 15 of the remaining 30 were kept after expert rating. Kept items were easier than FCI items (median difficulty index 0.69 against 0.57) and less discriminating (0.45 against 0.62), and their scores correlated r = 0.79 with the FCI. Their distractors did not match student difficulties.
  • Why it matters for us: Expect about 1 usable AI-written concept question in 7, and expect them to be easier and less diagnostic than expert-written ones until wrong options are tuned on our students’ data.
  • Where we use it: AI authoring literature review, design rules 6 and 11, the heavy-review list and the “not yet possible” list.

Using Large Language Models for Automatic Item Generation: Development and Validation for TIMSS Grade 4. Do AI-written test items hold up when students take them?

Section titled “Using Large Language Models for Automatic Item Generation: Development and Validation for TIMSS Grade 4. Do AI-written test items hold up when students take them?”

Muszyński, Plisiecki, Pokropek and Żółtak 2025, IEA R&D final report. Download · Source

  • Who was studied: GPT-4o items in the style of TIMSS Grade 4 maths and science (in Polish), unedited, mixed into a field test beside real TIMSS items. Field-test sample size not extracted.
  • How: Validation study with several prompt set-ups, expert rating and a field test.
  • What it found: Expert-rated correctness of maths items was 95% with five examples in the prompt, against 94% for real TIMSS items and 85% with no examples. The field test showed difficulty off target, some items with low or negative discrimination, repetitive items, implausible or repeated distractors, and sometimes no or several correct answers.
  • Why it matters for us: An item can look correct to experts and still fail with students; only a pilot with students shows the gap.
  • Where we use it: AI authoring literature review, design rule 6.

When AI Evaluates Its Own Work: Validating Learner-Initiated, AI-Generated Physics Practice Problems. Are physics problems an AI writes on demand any good, and can AI judges tell?

Section titled “When AI Evaluates Its Own Work: Validating Learner-Initiated, AI-Generated Physics Practice Problems. Are physics problems an AI writes on demand any good, and can AI judges tell?”

Geisler and Kortemeyer 2026, Physical Review Physics Education Research. Download · Source

  • Who was studied: 34 ETH Zurich introductory physics students who generated 543 practice problems in a course chatbot.
  • How: Exploratory study comparing an expert’s labels with AI judges.
  • What it found: AI judges agreed with the expert on “the model’s solution is correct” at F1 0.83, but only 0.61 on “problem is solvable”, 0.59 on “clear” and 0.23 to 0.24 on difficulty. An example from GPT-5.4 Thinking was underspecified with no correct option.
  • Why it matters for us: An AI check can be trusted for solution correctness and units, not for whether a problem is solvable, clear or the right difficulty.
  • Where we use it: AI authoring literature review, design rule 4.

Findings from the First Shared Task on Automated Prediction of Difficulty and Response Time for Multiple-Choice Questions. Can software predict how hard a question is from its text?

Section titled “Findings from the First Shared Task on Automated Prediction of Difficulty and Response Time for Multiple-Choice Questions. Can software predict how hard a question is from its text?”

Yaneva, North, Baldwin, Ha, Rezayi, Zhou, Ray Choudhury, Harik and Clauser 2024, BEA 2024 workshop. Download · Source

  • Who was studied: 667 retired US medical licensing MCQs; 17 teams.
  • How: Shared task (an open competition): teams predicted each item’s difficulty from its text.
  • What it found: Best RMSE (average prediction error; lower is better) 0.299, against 0.311 for a baseline that predicts the mean.
  • Why it matters for us: Question difficulty has to come from our students’ answers, not from the question text.
  • Where we use it: AI authoring literature review, design rule 7 and the “not yet possible” list.

Can LLMs Estimate Student Struggles? Does an AI’s sense of what is hard match students’?

Section titled “Can LLMs Estimate Student Struggles? Does an AI’s sense of what is hard match students’?”

Li, Chen, Xiao, Chen, Jiao and Zhou 2025, arXiv preprint. Download · Source

  • Who was studied: No students directly. Over 20 models rating item difficulty on four datasets (USMLE, Cambridge English, SAT reading, SAT maths), compared with real difficulty.
  • How: Benchmark, including models role-playing students of a set proficiency.
  • What it found: Average Spearman correlation (a 0-1 measure of whether two rankings agree) with real difficulty was 0.28, and about 0.41 on SAT maths. Stronger models were not better.
  • Why it matters for us: The coach cannot rely on a model to say how hard a question will be for our students.
  • Where we use it: AI authoring literature review, design rule 7 and the “not yet possible” list.

Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations. Can a simulated classroom of AI students rank question difficulty?

Section titled “Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations. Can a simulated classroom of AI students rank question difficulty?”

Acquaye, Huang, Carpuat and Rudinger 2026, arXiv preprint. Download · Source

  • Who was studied: No students directly. Open models role-played classrooms of US students on NAEP maths items for grades 4, 8 and 12.
  • How: Method study: the simulated answers were fitted with IRT (item response theory, the standard model for estimating item difficulty from who gets it right) and compared with NAEP’s real item statistics.
  • What it found: Correlations with real item difficulty reached 0.75, 0.76 and 0.82. Weaker models (Gemma) predicted better than stronger ones. Asking models directly how hard an item is worked poorly.
  • Why it matters for us: A simulated class could give a rough first ranking of new items before a pilot, but it has not been tested on JEE-level items.
  • Where we use it: AI authoring literature review, design rule 7 and “Difficulty”.

Automated Approaches for Scoring Math Misunderstandings in Student Self-Explanations. Can models tell which misconception a student’s written explanation shows?

Section titled “Automated Approaches for Scoring Math Misunderstandings in Student Self-Explanations. Can models tell which misconception a student’s written explanation shows?”

Crossley, Rittle-Johnson, Adler, Burleigh, Durkin, King and Benner 2026, AIME-Con 2026 (conference paper). Download · Source

  • Who was studied: Over 52,000 written explanations from grades 4-8 students on 15 Eedi MCQs; 1,850 competing teams.
  • How: Kaggle competition: classify each explanation as correct, a misconception (and which one), or neither.
  • What it found: Top MAP@3 (a 0-1 score for whether the right label is among a model’s top three guesses) about 0.948, against a 0.80 baseline. Performance on items the models were not trained on was not tested.
  • Why it matters for us: Once we have labelled answers for an item, a model can diagnose students’ misconceptions from their working; this is diagnosis, not authoring.
  • Where we use it: AI authoring literature review, design rule 5.