How good is AI at writing lessons, questions, hints and worked solutions?

Frontier models now solve most JEE Advanced problems, but not reliably enough to publish their solutions unchecked: on the held-out 2025 paper the best models answered 73% to 80% of questions correctly at the first try, and almost never fixed their own mistakes when asked to (Mukherjee and Ghosh 2025). Worked solutions and step hints can be used with light review when the solution is anchored to a verified key and several independent samples agree: that combination cut ChatGPT’s hint error rate from 32% to near 0% in algebra (Pardos and Bhandari 2024), and a tutor model given the item, the wrong answer and the misconception made 5 factual errors in 3,617 messages (LearnLM Team and Eedi 2025). New questions need heavy review: only 15 of 100 generated physics concept items survived expert screening (Küchemann et al. 2024), and generated items had more double-correct answers and give-away distractors than human ones (Doughty et al. 2024; Muszyński et al. 2025). Two things AI cannot yet do without Avanti’s own student data are write distractors that match real misconceptions and predict how hard an item will be (Feng et al. 2024; Fernandez et al. 2024; Yaneva et al. 2024; Li et al. 2025). For teachers, AI lesson preparation saves about 25 to 49 minutes a week with no loss of quality, in English primary and lower-secondary schools (Roy et al. 2024; Roy et al. 2026).

Scope note. No opened study tests AI-written content on Indian students, in JEE preparation, or in Hindi or Marathi physics. The two JEE studies test models solving problems, not students learning from model output. Every learning effect below comes from another population and should be read as a direction.

Evidence table

Only sources opened and read in the archived copy are listed.

Study	Design and population	n	Effect (unit)	What it means for the coach
Arora, Singh and Mausam 2023 (JEEBench)	Benchmark: GPT-4 (2023) and older models solving past JEE Advanced physics, chemistry and maths problems	515 problems	Best score 38.9% (GPT-4, chain of thought, 8-sample self-consistency). In 100 checked solutions: 34 conceptual, 30 computation, 15 grounding errors. Self-critique lowered the score from 35.0% to 33.9%; the verifier fixed 2 of 80 wrong solutions	2023 models could not author JEE Advanced solutions. Self-critique by the same model is not a check
Mukherjee and Ghosh 2025 (mmJEE-Eval)	Benchmark: 17 vision-language models (July-Oct 2025 APIs) on JEE Advanced 2019-2025, English and Hindi, official marking	1,460 questions	Held-out 2025 set, first-try accuracy: o3 72.7%, GPT-5 79.5%, Gemini 2.5 Flash 79.8%. GPT-5 scored 291/360 marks vs the topper’s 332. Models detected their own errors in 52.9% of cases and corrected 8% of those; GPT-5 chained detect-and-fix 5.2%. Hindi: GPT-5 right in English but wrong in Hindi on 7.3% of items	Current models solve about 4 in 5 JEE Advanced items. A generated solution must be checked against a key, never by asking the model to recheck. Hindi adds errors. Claude was not tested
Kortemeyer 2023	Case study: ChatGPT (Jan 2023) worked through a calculus-based introductory physics course at ETH Zurich, graded like a student	1 course: FCI, 76 homework problems, exams	FCI 60%; homework 55% (1.88 attempts); exams 47%; course grade 1.5, a narrow pass. Typical errors were plug-and-chug and arithmetic	Historical baseline only; shows the error types a reviewer should look for
Kieser et al. 2023	Lab study: ChatGPT on GPT-4 answering the Force Concept Inventory, as itself and as simulated students	Repeated runs of 30 FCI items	Averaged 83% on the FCI. Prompting it as a student of a given cohort produced no variance; prompting it to hold a named preconception produced varied answers that resembled real students in some respects	A model can role-play a named misconception, which is useful for piloting distractors. It cannot role-play “a weaker student” on request
Pardos and Bhandari 2024	Online RCT, 3 hint conditions x 4 maths topics (algebra, statistics), US Mechanical Turk adults; ChatGPT 3.5	274 learners; hints for 74 problems checked	32% of ChatGPT worked-solution hints failed a correctness check (25% to 47% by topic). With 10 samples and majority vote, errors fell to 0% to 2% in algebra and 13% in statistics. Checking took 37.6 seconds per hint. Learning gain: ChatGPT hints +17.0 pp, human hints +11.6 pp, no hints +1.9 pp; ChatGPT vs human not significant	Generated worked solutions can teach as well as human ones once wrong ones are removed. Self-consistency is the cheapest filter. Adults on easy maths, far from JEE
LearnLM Team and Eedi 2025	Exploratory RCT, five UK secondary schools, Years 9-10 maths; tutor drafts reviewed by a human tutor before sending	165 students; 3,617 drafted messages	74.4% of drafts sent with no edit, 76.4% with no or 1-2 character edits; 5 factual errors (0.1%); 44.3% of edits slowed pacing, 19.5% changed tone. Transfer to a new topic 66.2% vs 60.7% for human tutors (+5.5 pp, interval -1.4 to +12.4)	A model given the item, the wrong answer and the misconception writes accurate step guidance. Human edits were mostly about pacing and tone, not physics. Written by the model’s developer
Scarlatos et al. 2024	Method study, Eedi middle-school maths MCQs with teacher-written feedback for each wrong option	1,956 questions; 80 messages hand-checked	GPT-4 as a rubric judge matched human labels on “feedback is correct” with accuracy 0.77 (F1 0.81); two humans agreed at kappa 0.53. Teacher-written feedback was itself rated correct 91% of the time	An LLM judge catches most but not all wrong feedback. Even human-written feedback needs checking
Feng et al. 2024	Method study, Eedi maths MCQs (“Number” topic), several LLMs and prompts; 2 rater graduate students	1.4K MCQs; 20 rated	Best method (retrieve similar items as examples) matched all three teacher distractors on 11% of items and at least one on 74%. Raters scored LLM distractors lower: validity 3.28 vs 3.99, plausibility 2.68 vs 3.72 (1-5 scale)	LLMs write mathematically possible wrong answers, not the ones students actually pick. Retrieval of Avanti’s own similar items is the best cheap method
Fernandez et al. 2024 (DiVERT)	Method study, Eedi maths MCQs; a 7B open model fine-tuned on teacher error labels vs GPT-4o	1,434 MCQs; 180 errors rated by 2 teachers	Teacher-rated error quality: human 3.23, fine-tuned 7B 3.07 (not significantly different), GPT-4o 2.56 (significantly worse). Rater agreement was low (r = 0.33)	Fine-tuning a small model on labelled student errors beats a frontier model at naming misconceptions. This is the route Purna proposed, and it needs Avanti’s labelled wrong-answer data
Bitew et al. 2023	Method study, secondary-school MCQs from the WeZooz test set (language learning and factual items in sciences, geography, history, biology); teacher ratings	300 questions, 10 teachers	53% of ChatGPT distractors (prompted with retrieved similar items) rated usable as is	About half of generated distractors are usable in factual subjects. Far from JEE problem solving
Doughty et al. 2024	Comparative study, GPT-4 vs human MCQs for 6 university Python courses, rubric annotation	651 generated, 449 human MCQs; 246 objectives	Multiple correct answers: 4.9% generated vs 1.1% human. Distractor gives the answer away: 4.0% vs 0.9%. Generated items were better aligned to learning objectives	Generated MCQs need a check for a unique key and for giveaway options. Errors cluster in higher-order (apply, create) items
Küchemann et al. 2024	Item development study: ChatGPT 3.5 concept items in kinematics and forces, expert rating, then given with the FCI to first-year university students	100 generated, 15 kept; 172 students	70 of 100 removed at first screen (unrealistic, duplicate, no or two correct answers); 15 of 30 kept after expert rating. Kept items: median difficulty index 0.69 vs FCI 0.57 (easier), discrimination 0.45 vs 0.62; scores correlated r = 0.79 with the FCI. Distractors did not match student difficulties	Purna-style conceptual items are possible, with about a 15% yield and expert selection. Without student data they are easier and less diagnostic than expert items
Muszyński et al. 2025 (IEA)	Validation study: GPT-4o items in the style of TIMSS Grade 4 maths and science (Polish), unedited, expert-rated and field-tested beside real TIMSS items	Several prompt setups, mixed into a field test	Maths items: expert-rated correctness 95% (five examples in prompt) vs 94% for TIMSS items, 85% with no examples. Field test: difficulty off-target, some items with low or negative discrimination, repetitive items, implausible or repeated distractors, sometimes no or several correct answers	Examples in the prompt matter. Correctness can match human items while psychometric quality does not. Only a field test shows the gap
Geisler and Kortemeyer 2026	Exploratory study, ETH Zurich introductory physics; students generated practice problems on demand in a course chatbot; expert labels vs LLM judges	34 students, 543 problems	LLM judges agreed with the expert on “the model’s solution is correct” at F1 0.83, but only 0.61 on “problem is solvable”, 0.59 on “clear”, and 0.23 to 0.24 on difficulty. A GPT-5.4 Thinking example was underspecified with no correct option	On-demand generated physics problems are often flawed. Automated checks work for solution correctness and units, not for solvability or difficulty
Yaneva et al. 2024 (BEA shared task)	Shared task: predict difficulty of retired US medical licensing MCQs from text	667 items, 17 teams	Best RMSE 0.299 vs 0.311 for a baseline that predicts the mean	Predicting difficulty from item text barely beats guessing. Difficulty must come from student data
Li et al. 2025	Benchmark: over 20 LLMs asked to rate item difficulty, incl. role-playing students of set proficiency (USMLE, Cambridge English, SAT reading, SAT maths)	4 datasets	Average Spearman correlation with real difficulty 0.28; about 0.41 on SAT maths. Stronger models were not better	A model’s own sense of “hard” does not match students’
Acquaye et al. 2026	Method study: open LLMs role-play classrooms of US students; simulated answers fitted with IRT and compared with NAEP maths item statistics	NAEP grades 4, 8, 12	Correlations with real item difficulty up to 0.75, 0.76 and 0.82. Weaker models (Gemma) predicted better than stronger ones. Direct difficulty judgments were poor	Simulated classrooms can give a rough prior on difficulty before a pilot. Not tested on JEE-level items
Crossley et al. 2026	Kaggle competition: classify students’ written explanations for Eedi MCQs as correct, misconception (which one), or neither	52,000+ explanations on 15 MCQs, grades 4-8; 1,850 teams	Top MAP@3 about 0.948 vs a 0.80 baseline	When trained on labelled answers to the same items, models diagnose misconceptions well. This is diagnosis, not authoring; performance on items the models were not trained on was not tested
Gupta et al. 2025	Benchmark: 6 LLMs (incl. Claude 3.7 Sonnet, GPT-4o) on Eedi-based educational tasks in English and 8 languages incl. Hindi and Telugu, machine-translated	4 tasks	Misconception identification, GPT-4o: 97.6% English vs 94.5% Hindi. Feedback selection: 53.4% vs 48.7%, and 32.1% when the prompt was also in Hindi. Average across tasks: English 70.9%, Hindi 63.1%	Hindi output loses accuracy, more with Hindi prompts. Keep authoring prompts in English and check Hindi output
Roy et al. 2024 (EEF)	Cluster RCT, Year 7-8 science teachers in England, ChatGPT plus a guide	259 teachers, 68 schools	Preparation 56.2 vs 81.5 min/week (25.3 min saved); no quality difference on blind expert review	AI saves teacher preparation time when the teacher adapts the output
Roy et al. 2026 (EEF, Aila)	Cluster RCT, Years 3-6 teachers in England, Oak’s curriculum-grounded lesson assistant, 10 weeks in autumn 2025	464 teachers, 108 schools	Preparation 2.5 vs 3.3 h/week (48.6 min saved; ratio 76%, range 64% to 89%); resource quality no worse. Used for 7% of lessons; fewer than 1 in 5 said it changed their teaching	Grounding in a quality-assured bank keeps quality; savings come with light, selective use
Clark et al. 2024 (Oak auto-evaluation)	Method study: LLM judge (GPT-4o) scoring Aila lessons on 24 benchmarks, compared with Oak teachers on quiz distractor quality	4,985 lessons; 311 MCQs rated by 19 teachers	Before tuning, the judge scored lower than teachers on 75% of quizzes; QWK 0.17. After adding teacher-derived examples, QWK 0.32 and lower on 62%	An LLM judge for distractor quality needs calibration against teachers and stays only moderately aligned. Written by the tool’s developer

What the evidence says

Can current models solve JEE problems reliably?

Not reliably enough to publish without a check. In 2023 GPT-4 scored under 40% on JEE Advanced problems even with chain-of-thought and self-consistency; its errors were missing concepts, arithmetic, and wrong translation of concepts into equations (Arora et al. 2023). By 2025 frontier models answered 72.7% to 79.8% of held-out JEE Advanced 2025 questions correctly at the first try, and Gemini 2.5 Flash and GPT-5 scored 300 and 291 of 360 marks, against 332 for the topper (Mukherjee and Ghosh 2025). Open models scored 37% to 45%. For conceptual physics, GPT-4 averaged 83% on the Force Concept Inventory (Kieser et al. 2023), up from 60% for the January 2023 ChatGPT (Kortemeyer 2023).

A 20% error rate per item is too high for a solution a student will study. Sampling helps: GPT-5 reached 100% within three attempts in the error-correction experiment, and three samples raised scores by 30.4% on average (Mukherjee and Ghosh 2025). Self-review does not: models found errors in about half their wrong answers but fixed 8% of those (Mukherjee and Ghosh 2025), and GPT-4’s self-critique lowered its JEEBench score (Arora et al. 2023). Claude was not in either benchmark, so Avanti would need to measure its chosen model on its own bank.

Hints, worked solutions and step feedback

Generated worked solutions can teach. Adults who received ChatGPT worked-solution hints on algebra and statistics gained 17.0 points, against 11.6 for human-written hints, and the difference was not significant (Pardos and Bhandari 2024). But 32% of the raw hints had a wrong answer or wrong steps. Taking the majority answer of 10 samples cut errors to 0% to 2% in algebra and 13% in statistics, and a human check took under 40 seconds per hint (Pardos and Bhandari 2024).

Grounding matters most. LearnLM was given the question, the student’s wrong answer and the misconception behind it. Supervising tutors sent 74.4% of its messages unchanged and found 5 factual errors in 3,617 (LearnLM Team and Eedi 2025). Most edits slowed the pace or warmed the tone, not the content. For short feedback on a wrong option, GPT-4’s feedback rated highly on a rubric, but the rubric judge was also GPT-4, and the authors warn this may inflate the score (Scarlatos et al. 2024).

Questions and distractors

Generated questions are usable after heavy filtering. From 100 ChatGPT 3.5 concept items on forces, 70 failed a first screen and 15 survived expert rating; the survivors were easier and less discriminating than FCI items (Küchemann et al. 2024). GPT-4 programming MCQs were mostly sound, but 4.9% had more than one correct answer and 4.0% had a distractor that gave the answer away, several times the human rate (Doughty et al. 2024). GPT-4o maths items in TIMSS style were rated as correct as real TIMSS items when the prompt carried five examples, but a field test showed off-target difficulty, weak discrimination and repetitive items (Muszyński et al. 2025). Physics problems generated on demand for exam revision were often underspecified or unsolvable (Geisler and Kortemeyer 2026).

Distractors are the weakest part. LLM distractors are often mathematically possible but not what students choose: raters scored their plausibility 2.68 against 3.72 for teacher distractors (Feng et al. 2024), and GPT-4o’s error descriptions scored 2.56 against 3.23 for teachers’ (Fernandez et al. 2024). The fix in both studies was Avanti-style data: retrieving similar items with known distractors as examples (Feng et al. 2024), or fine-tuning a small model on teacher-labelled errors, which matched human quality (Fernandez et al. 2024). In factual subjects about half of generated distractors were usable as is (Bitew et al. 2023). Once labelled answers exist for an item, models classify students’ misconceptions from their written explanations very accurately (Crossley et al. 2026).

Difficulty

Models cannot tell how hard an item is for students. Text-based prediction of item difficulty barely beat predicting the average (Yaneva et al. 2024). LLM difficulty ratings correlated 0.28 with real difficulty on average, and stronger models did not do better (Li et al. 2025). LLM judges agreed with an expert on difficulty at F1 0.23 to 0.24 (Geisler and Kortemeyer 2026). One promising method simulates a classroom of students at several skill levels with a weaker model and fits an IRT model, reaching correlations of 0.75 to 0.82 with NAEP item statistics (Acquaye et al. 2026). A model told to answer as “a weaker student” still answers like itself, while a model told to hold a named misconception does vary (Kieser et al. 2023).

Lesson plans and teacher time

Two high-security EEF trials found the same pattern. ChatGPT cut secondary science teachers’ preparation from 81.5 to 56.2 minutes a week (Roy et al. 2024). Oak’s Aila cut primary teachers’ preparation from 3.3 to 2.5 hours a week, with resource quality no worse, though it was used for only 7% of lessons and teachers often adapted its output (Roy et al. 2026). Oak checks Aila’s lessons with an LLM judge on 24 benchmarks, but the judge’s agreement with teachers on quiz quality was low (QWK 0.17) and rose only to 0.32 after tuning (Clark et al. 2024). In both trials a teacher stood between the AI and the pupil.

Indian languages

Accuracy drops in Hindi. On JEE Advanced 2025, GPT-5 was right in English and wrong in Hindi on 7.3% of items, and right only in Hindi on 3.7% (Mukherjee and Ghosh 2025). On Eedi tasks translated to Hindi, GPT-4o lost about 3 to 5 points with English prompts and 21 points on feedback selection with Hindi prompts (Gupta et al. 2025). Part of that drop may come from machine translation of the test items, which scored lowest for Hindi among the eight languages. No opened study tests generated physics lessons in Hindi or Marathi.

Answer: what AI can author today

Light review (sampled audit) is enough for:

• Worked solutions and step hints for an item that has a verified key, when several independent samples agree with the key (Pardos and Bhandari 2024; Mukherjee and Ghosh 2025).
• Step feedback for a known wrong answer when the model is given the misconception (LearnLM Team and Eedi 2025).
• Teacher-facing lesson outlines that a teacher adapts before use (Roy et al. 2024; Roy et al. 2026).

Heavy review (every item, by a subject expert) is needed for:

• New questions of any kind, including ConcepTests and lesson ladders. Expect to discard most drafts: 15 of 100 concept items survived in one physics study, and about half of generated distractors were usable in factual subjects (Küchemann et al. 2024; Bitew et al. 2023).
• Solutions to items with no verified key, especially multi-step JEE Advanced problems (Mukherjee and Ghosh 2025).
• Any student-facing Hindi or Marathi content (Gupta et al. 2025).

Not yet possible without Avanti’s own student data:

• Distractors that match the mistakes Avanti’s students make (Feng et al. 2024; Fernandez et al. 2024; Küchemann et al. 2024).
• Knowing an item’s difficulty before students try it (Yaneva et al. 2024; Li et al. 2025).
• A model checking its own work (Arora et al. 2023; Mukherjee and Ghosh 2025).

Where it is weak or contested

• No Indian students, no JEE learners. The JEE studies test models solving problems. The learning effects come from US adults (Pardos and Bhandari 2024) and UK 13 to 15 year olds (LearnLM Team and Eedi 2025).
• Old models. Several generation studies used ChatGPT 3.5 or GPT-4 (Pardos and Bhandari 2024; Küchemann et al. 2024; Feng et al. 2024). Error rates are likely lower today, but the 2026 physics study still shows a recent model writing an unsolvable problem (Geisler and Kortemeyer 2026). Distractor plausibility has not been shown to improve with model size.
• Developer-written evaluations. LearnLM (Google and Eedi) and Oak’s auto-evaluation (Oak) report on their own tools.
• Small rater pools. Distractor ratings rest on two raters per study with low agreement (r = 0.33 in Fernandez et al. 2024; QWK 0.23 to 0.54 in Feng et al. 2024). Even humans disagree about what makes a good distractor.
• LLM judges grading LLM output. Scarlatos et al. (2024) used GPT-4 to grade GPT-4 feedback and flag that this may inflate scores.
• Lesson quality is teacher-mediated. The EEF trials measured lessons a teacher prepared with AI help, not AI-only lessons sent to students (Roy et al. 2024; Roy et al. 2026).
• Eedi’s Kaggle misconception competition (2024-25). Eedi’s blog describes the winning method (Qwen retrieval and reranking, with Claude-written student rationales) but gives no accuracy figure, so no result is cited from it (Eedi 2025).

Design rules for an Avanti lesson

1. Build every lesson on a bank item with a verified key. The ladder from a hard problem down to its steps starts from Avanti’s own key and stepped solution. The model writes around it; it does not supply the answer (LearnLM Team and Eedi 2025; Pardos and Bhandari 2024).
2. Check each generated step by independent re-solving, not self-review. Generate the solution 8 to 10 times and accept a step sequence only when the majority answer matches the key. Route disagreements to a teacher (Pardos and Bhandari 2024; Arora et al. 2023; Mukherjee and Ghosh 2025).
3. Never ask the authoring model to check its own work. Self-critique lowered scores and fixed almost nothing (Arora et al. 2023; Mukherjee and Ghosh 2025).
4. Use an LLM judge only where it has been shown to agree with experts. It is acceptable for “the solution is correct”, units, and “the answer is not in the problem” (F1 about 0.83). It is not acceptable for solvability, clarity or difficulty (Geisler and Kortemeyer 2026). Calibrate any judge against teachers first (Clark et al. 2024).
5. Take distractors and feedback from Avanti’s own wrong answers. Use the options students actually chose on the new CMS, retrieve similar items as examples, and let the model name the misconception for a teacher to approve (Feng et al. 2024; Fernandez et al. 2024; Crossley et al. 2026).
6. Every new item is reviewed by a subject teacher before any student sees it. Plan for most drafts to be rejected (Küchemann et al. 2024; Doughty et al. 2024). Check specifically for a unique correct answer and for options that give the answer away (Doughty et al. 2024; Muszyński et al. 2025).
7. Measure difficulty with students, not with the model. Pilot new items in low-stakes practice and set difficulty from the response data. A simulated classroom can rank items before a pilot, but is not proof (Yaneva et al. 2024; Li et al. 2025; Acquaye et al. 2026).
8. Track the review, not an approval target. Log teacher approve, edit and reject decisions and the reason, as LearnLM did. Move a content type to sampled audit only after its error rate is measured as low (LearnLM Team and Eedi 2025).
9. Measure the chosen model on Avanti’s bank before trusting it. Run it on held-back items per chapter and record its error rate; published JEE results do not include Claude (Mukherjee and Ghosh 2025).
10. Author in English; check Hindi and Marathi by hand. Keep authoring prompts in English and have a teacher read any translated lesson (Gupta et al. 2025; Mukherjee and Ghosh 2025).
11. For Purna-style concept items, budget for selection and a field test. The 2023 idea (generate items linked to an anchor, expert-verify, use for peer instruction) is feasible, but expect about 1 usable item in 7 and items easier than expert ones until distractors are tuned on student data (Küchemann et al. 2024; Fernandez et al. 2024).
12. Expect teacher time savings near half an hour a week, not more. That is the size found in both EEF trials, with a teacher adapting the output (Roy et al. 2024; Roy et al. 2026).

References

• Archived copies, with a .txt extraction and SHA-256 sums, are in gs://avantifellows-research-sources/ai-coach/ai_authoring/ (Avanti Google login); see SOURCES.md.

• Acquaye, C., Huang, Y. T., Carpuat, M., and Rudinger, R. (2026). Take out your calculators: Estimating the real difficulty of question items with LLM student simulations. arXiv:2601.09953 (v2, 20 Apr 2026). https://arxiv.org/abs/2601.09953 Opened 2026-10-30. File: takeout2026_difficulty_sim.pdf.

• Arora, D., Singh, H. G., and Mausam (2023). Have LLMs advanced enough? A challenging problem solving benchmark for large language models. arXiv:2305.15074 (v3, 23 Oct 2023). https://arxiv.org/abs/2305.15074 Opened 2026-10-30. File: arora2023_jeebench.pdf.

• Bitew, S. K., Deleu, J., Develder, C., and Demeester, T. (2023). Distractor generation for multiple-choice questions with predictive prompting and large language models. arXiv:2307.16338. https://arxiv.org/abs/2307.16338 Opened 2026-10-30. File: bitew2023_distractors.pdf.

• Clark, H.-B., Dowland, M., Benton, L., Budai, R., Keskin, I. K., Searle, E., Gregory, M., Hodierne, M., Gayne, W., and Roberts, J. (2024). Auto-evaluation: A critical measure in driving improvements in quality and safety of AI-generated lesson resources. MIT Open Learning, AI + Open Education Initiative, 26 Nov 2024. https://aiopeneducation.pubpub.org/pub/i36sncz8 (also arXiv:2502.10410). Opened 2026-10-30. File: clark2025_oak_autoeval.pdf.

• Crossley, S. A., Rittle-Johnson, B., Adler, R., Burleigh, L., Durkin, K., King, J., and Benner, M. (2026). Automated approaches for scoring math misunderstandings in student self-explanations. Proceedings of AIME-Con 2026, Vol. 1, pp. 297-302. https://aclanthology.org/2026.aimecon-main.33.pdf Opened 2026-10-30. File: aimecon2026_33.pdf.

• Doughty, J., Wan, Z., Bompelli, A., et al. (2024). A comparative study of AI-generated (GPT-4) and human-crafted MCQs in programming education. ACE 2024. https://doi.org/10.1145/3636243.3636256 (arXiv:2312.03173). Opened 2026-10-30. File: doughty2024_mcq_gpt4.pdf.

• Eedi (2025). From wrong answers to real insights: How we used a Kaggle challenge to map student misconceptions. Blog post. https://www.eedi.com/news/from-wrong-answers-to-real-insights-how-we-used-a-kaggle-challenge-to-map-student-misconceptions Opened 2026-10-30. File: eedi2025_kaggle_blog.html.

• Feng, W., Lee, J., McNichols, H., Scarlatos, A., Smith, D., Woodhead, S., Otero Ornelas, N., and Lan, A. (2024). Exploring automated distractor generation for math multiple-choice questions via large language models. arXiv:2404.02124 (v3, 18 Apr 2024). https://arxiv.org/abs/2404.02124 Opened 2026-10-30. File: feng2024_distractors.pdf.

• Fernandez, N., Scarlatos, A., Feng, W., Woodhead, S., and Lan, A. (2024). DiVERT: Distractor generation with variational errors represented as text for math multiple-choice questions. arXiv:2406.19356 (v2, 8 Oct 2024). https://arxiv.org/abs/2406.19356 Opened 2026-10-30. File: fernandez2024_divert.pdf.

• Geisler, T., and Kortemeyer, G. (2026). When AI evaluates its own work: Validating learner-initiated, AI-generated physics practice problems. Physical Review Physics Education Research 22, 010134. https://doi.org/10.1103/9hh9-vt4d Opened 2026-10-30. File: prper2026_9hh9.pdf.

• Gupta, V., Pal Chowdhury, S., Zouhar, V., Rooein, D., and Sachan, M. (2025). Are large language models for education reliable for all languages? arXiv:2504.17720 (v2, 5 Aug 2025). https://arxiv.org/abs/2504.17720 Opened 2026-10-30. File: eth2025_multilingual_edu_tasks.pdf.

• Kieser, F., Wulff, P., Kuhn, J., and Küchemann, S. (2023). Educational data augmentation in physics education research using ChatGPT. Physical Review Physics Education Research 19, 020150. https://doi.org/10.1103/PhysRevPhysEducRes.19.020150 Opened 2026-10-30. File: kieser2023_prper_fci_synthetic.pdf.

• Kortemeyer, G. (2023). Could an artificial-intelligence agent pass an introductory physics course? arXiv:2301.12127 (v2, 1 Feb 2023). https://arxiv.org/abs/2301.12127 Opened 2026-10-30. File: kortemeyer2023_physics_course.pdf.

• Küchemann, S., Rau, M., Schmidt, A., and Kuhn, J. (2024). ChatGPT’s quality: Reliability and validity of concept inventory items. Frontiers in Psychology 15: 1426209. https://doi.org/10.3389/fpsyg.2024.1426209 Opened 2026-10-30. File: kuchemann2024_frontiers_ci_items.pdf.

• LearnLM Team, Google, and Eedi (2025). AI tutoring can safely and effectively support students: An exploratory RCT in UK classrooms. arXiv:2512.23633. https://arxiv.org/abs/2512.23633 Opened 2026-10-30 (copied from the ai_tutors strand). File: learnlm2025_eedi_rct.pdf.

• Li, M., Chen, H., Xiao, Y., Chen, J., Jiao, H., and Zhou, T. (2025). Can LLMs estimate student struggles? Human-AI difficulty alignment with proficiency simulation for item difficulty prediction. arXiv:2512.18880 (v2, 10 May 2026). https://arxiv.org/abs/2512.18880 Opened 2026-10-30. File: misalign2025_difficulty.pdf.

• Mukherjee, A., and Ghosh, S. (2025). mmJEE-Eval: A bilingual multimodal benchmark for evaluating scientific reasoning in vision-language models. arXiv:2511.09339 (v1, 12 Nov 2025). https://arxiv.org/abs/2511.09339 Opened 2026-10-30. File: mukherjee2025_mmjee.pdf.

• Muszyński, M., Plisiecki, H., Pokropek, A., and Żółtak, T. (2025). Using large language models for automatic item generation: Development and validation for TIMSS Grade 4. Final report for the IEA R&D 2023 research project, IFiS PAN; posted by IEA October 2025. https://www.iea.nl/sites/default/files/2025-10/Muszy%C5%84ski%20AIG%20LLM%20Report.pdf Opened 2026-10-30. File: muszynski2025_iea_aig.pdf.

• Pardos, Z. A., and Bhandari, S. (2024). ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills. PLOS ONE 19(5): e0304013. https://doi.org/10.1371/journal.pone.0304013 Opened 2026-10-30 (copied from the ai_tutors strand). File: pardos2024_chatgpt_hints.pdf.

• Roy, P., Poet, H., Staunton, R., Aston, K., and Thomas, D. (2024). ChatGPT in lesson preparation: A Teacher Choices trial. Evaluation report. Education Endowment Foundation / NFER. https://files.eric.ed.gov/fulltext/ED680921.pdf Opened 2026-10-30 (copied from the hybrid_human_ai strand). File: roy2024_eef_chatgpt_lesson_prep.pdf.

• Roy, P., Poet, H., Staunton, R., Aston, K., Thomas, D., and Styles, B. (2026). Lesson planning using AI lesson assistant, Aila: A Teacher Choices trial. Evaluation report, October 2026. Education Endowment Foundation / NFER. https://d2tic4wvo1iusb.cloudfront.net/production/documents/projects/aila-lesson-planning_teacher-choices_evaluation-report.pdf (link found via the Wayback copy of the EEF project page, which blocked direct access). Opened 2026-10-30. File: eef2026_aila_trial.pdf.

• Scarlatos, A., Smith, D., Woodhead, S., and Lan, A. (2024). Improving the validity of automatically generated feedback via reinforcement learning. AIED 2024. arXiv:2403.01304. https://arxiv.org/abs/2403.01304 Opened 2026-10-30. File: scarlatos2024_feedback_rl.pdf.

• Yaneva, V., North, K., Baldwin, P., Ha, L. A., Rezayi, S., Zhou, Y., Ray Choudhury, S., Harik, P., and Clauser, B. (2024). Findings from the first shared task on automated prediction of difficulty and response time for multiple-choice questions. BEA 2024. https://aclanthology.org/2024.bea-1.39.pdf Opened 2026-10-30. File: yaneva2024_bea_difficulty.pdf.

Leads, not opened

These were found but not read in full. Their findings are not stated as fact above.

• ByteDance Seed technical report on LLMs taking JEE Advanced 2025 (press coverage reports Gemini 2.5 Pro above the topper’s 332/360). Searched for the primary report; found only news and blog coverage. Not cited.
• Kaggle, Eedi “Mining Misconceptions in Mathematics” competition leaderboard and winning write-ups (2024-25). Kaggle pages not retrieved; a 13th-place post reports private MAP@25 of 0.561, seen only in a search snippet.
• ImpactEd’s 2024-25 evaluation of Oak National Academy (teacher workload and pupil outcomes). Seen only in search results; not retrieved.
• “Synthetic student responses: LLM-extracted features for IRT difficulty parameter estimation” (2026). arXiv:2602.00034. Authors not checked. A reported correlation of 0.85 was seen only in a search summary.
• “IndicEval: A bilingual Indian educational evaluation framework for large language models” (2026). arXiv:2602.16467. Authors not checked. Reported to test UPSC, JEE and NEET questions in English and Hindi. Not downloaded; the most relevant open lead for Hindi.
• Maity, S., Deroy, A., and Sarkar, S. (2024). Exploring the capabilities of prompted LLMs in educational and assessment applications (NCERT-based question generation incl. Hindi). arXiv:2405.11579. Skimmed, not read in enough depth to cite, and not kept in the archive; uses 2023 models.
• Singh, H., Gupta, N., Bharadwaj, S., Tewari, D., and Talukdar, P. (2024). IndicGenBench. arXiv:2404.16816. Skimmed; covers general generation tasks (summarisation, translation, QA), not educational content. Not cited and not kept in the archive.