Teachers, tutoring software and adaptive platforms
Part of the AI coach reading library: 42 papers cited in the AI coach notes, each in plain language. Who was studied, how, what it found, why it matters for our students, and where our notes use it. Teachers and AI working together, coaching and nudges, step-based tutoring, and adaptive platforms such as Mindspark, Khan Academy and ASSISTments. Download links open the archived copy and need an Avanti Google login; the Source link is the public original.
The overviews are drawn from the extraction tables and literature reviews written when each paper was read, not from a fresh reading. Where a number here and in a note differ, the note’s section is the one to check.
Teachers, tutors and AI working together
Section titled “Teachers, tutors and AI working together”Not Too Late: Improving Academic Outcomes Among Adolescents. Does daily small-group tutoring raise high-school maths scores, and is it the relationship that does it?
Section titled “Not Too Late: Improving Academic Outcomes Among Adolescents. Does daily small-group tutoring raise high-school maths scores, and is it the relationship that does it?”Guryan et al. 2021, NBER Working Paper 28531. Download · Source
- Who was studied: Grade 9-10 students in Chicago public schools: 2,633 students in one trial and 2,710 in the other.
- How: Two randomised controlled trials (RCTs: students were assigned to the programme or not by chance). Tutoring was two students to one tutor, one class period every day (Saga tutoring).
- What it found: Maths scores rose by 0.16 SD in the first trial and 0.37 SD in the second (SD: standard deviations, the usual unit for comparing test-score gains). There was no detectable change in students feeling supported by adults, grit, conscientiousness or locus of control, so the authors conclude the gain came from the instruction, not a “mentoring effect”.
- Why it matters for us: Gains came from daily, structured instruction, so the coach’s dosage has to come from frequent practice sessions, not from the relationship with a teacher.
- Where we use it: Hybrid human-AI literature review (learning gains come from dosage and instruction; design rule 2 on spreading sessions across the week).
Improving Student Learning with Hybrid Human-AI Tutoring: A Three-Study Quasi-Experimental Investigation. Does adding remote human tutors to maths software help, and how should their time be shared?
Section titled “Improving Student Learning with Hybrid Human-AI Tutoring: A Three-Study Quasi-Experimental Investigation. Does adding remote human tutors to maths software help, and how should their time be shared?”Thomas et al. 2024, LAK ’24 conference paper. Download · Source
- Who was studied: Students in three low-income US middle schools: 125, 385 and 75 students in the three studies.
- How: Quasi-experiment (groups compared without random assignment). Remote human tutors worked alongside maths software.
- What it found: Hybrid tutoring increased time on the software and skills mastered. A round-robin rule giving every student equal tutor time may have slowed students at or above grade level. Moving from one tutor per 8 students to one per 4 raised lessons completed per hour (+0.36 workspaces per hour).
- Why it matters for us: Teacher help in a coach session should go to the students who are stuck, not be shared out equally by rotation.
- Where we use it: Hybrid human-AI literature review (design rule 4, route teacher attention by need; spec conflict on the rotation floor).
Student Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms. Do live analytics for teachers raise learning in a tutoring-software classroom?
Section titled “Student Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms. Do live analytics for teachers raise learning in a tutoring-software classroom?”Holstein, McLaren and Aleven 2018, AIED 2018 conference paper. Download · Source
- Who was studied: 286 US middle-school students in 18 maths classes with 8 teachers, using an intelligent tutor (software that gives feedback on each step).
- How: Classroom RCT with three arms. Teachers wore smart glasses (Lumilo) that showed live analytics on each student, or a monitoring-only version, or nothing.
- What it found: Analytics beat business-as-usual on learning gain (r = 0.21, an effect-size measure on a correlation scale) and beat monitoring alone (r = 0.11). Monitoring alone also beat business-as-usual (r = 0.13); the authors suggest students who knew they were watched gamed the software less. The analytics moved teacher time towards students with low prior knowledge and narrowed the gap between weaker and stronger students.
- Why it matters for us: A teacher view that says who is stuck helps, and telling students that their teacher sees their work may itself help.
- Where we use it: Hybrid human-AI literature review (design rules 4 and 7; spec conflict on students feeling watched).
Designing for human-AI complementarity in K-12 education. What do teachers actually do when software flags a student?
Section titled “Designing for human-AI complementarity in K-12 education. What do teachers actually do when software flags a student?”Holstein and Aleven 2021, arXiv paper. Download · Source
- Who was studied: 14 teachers and 304 students in the design pilots for the Lumilo teacher glasses, plus the later trial.
- How: A design account of the pilots and trial. Descriptive only, with no comparison group.
- What it found: Teachers spent about 24 seconds per student visit, and often prompted reflection or encouraged rather than teaching content.
- Why it matters for us: Short teacher moves in response to an AI flag are a realistic unit of teacher work at a Centre.
- Where we use it: Hybrid human-AI literature review (evidence table; “AI that advises the human works, modestly”).
ChatGPT in Lesson Preparation: A Teacher Choices Trial. Does ChatGPT save teachers preparation time without lowering quality?
Section titled “ChatGPT in Lesson Preparation: A Teacher Choices Trial. Does ChatGPT save teachers preparation time without lowering quality?”Roy et al. 2024, Education Endowment Foundation / NFER evaluation report. Download · Source
- Who was studied: 259 Year 7-8 science teachers in 68 schools in England.
- How: RCT with schools assigned by chance. Teachers got ChatGPT and a short guide.
- What it found: Preparation took 56.2 minutes a week against 81.5 for controls, a saving of 25.3 minutes (31%), with no quality difference on blind expert review. Teachers used it for one or two activities, mostly writing questions and quizzes, and used it less as the trial went on. Student outcomes were not measured.
- Why it matters for us: About 25 minutes a week is the realistic saving to set against the time a teacher-review loop adds.
- Where we use it: Hybrid human-AI literature review (design rule 11, measure teacher time net); AI authoring literature review (teacher time saved).
How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Study. Does heavy chatbot use leave people lonelier?
Section titled “How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Study. Does heavy chatbot use leave people lonelier?”Fang et al. 2025, arXiv preprint. Download · Source
- Who was studied: 981 adults from a US online panel. No adolescents.
- How: Four-week RCT of different chatbot modes.
- What it found: The assigned condition had no effect. Heavier voluntary daily use was associated with more loneliness and emotional dependence, but use was not randomised, so this part is correlational.
- Why it matters for us: The coach should stay task-focused and route personal or distress content to a named adult, rather than become the student’s confidant.
- Where we use it: Hybrid human-AI literature review (what should stay human; design rule 10).
Coaching, mentoring, nudges and early warning
Section titled “Coaching, mentoring, nudges and early warning”The Effects of Student Coaching in College: An Evaluation of a Randomized Experiment in Student Mentoring. Does proactive professional coaching keep college students enrolled?
Section titled “The Effects of Student Coaching in College: An Evaluation of a Randomized Experiment in Student Mentoring. Does proactive professional coaching keep college students enrolled?”Bettinger and Baker 2011, NBER Working Paper 16881. Download · Source
- Who was studied: 13,555 US college students, mostly non-traditional.
- How: RCT. Professional coaches (InsideTrack) contacted students by phone, email and text about goals, time management and study skills.
- What it found: Enrolment rose 5.2 percentage points (pp) at 6 months and 3.4 pp at 24 months.
- Why it matters for us: Proactive coaching moves persistence, which is closer to what teacher one-on-ones can do than raising test scores.
- Where we use it: Hybrid human-AI literature review (coaching moves persistence more reliably than grades).
Student Coaching: How Far Can Technology Go? Do texts and online exercises work as well as a personal coach?
Section titled “Student Coaching: How Far Can Technology Go? Do texts and online exercises work as well as a personal coach?”Oreopoulos and Petronijevic 2016, NBER Working Paper 22630. Download · Source
- Who was studied: Over 4,000 students at a Canadian university (Toronto); the personal coaching arm had only 24 students.
- How: RCT comparing an online exercise, one-way text messages and personal coaching.
- What it found: Coaching raised GPA by 0.35 SD; texts and the online exercise had no effect. The authors credit coaches who started conversations, remembered earlier ones and built trust. The coaching estimate rests on a very small arm.
- Why it matters for us: Messages between sessions should be two-way and specific to the student’s task, not generic reminders.
- Where we use it: Hybrid human-AI literature review (coaching evidence; design rule 8).
The Remarkable Unresponsiveness of College Students to Nudging and What We Can Learn from It. Does coaching raise grades when run at scale?
Section titled “The Remarkable Unresponsiveness of College Students to Nudging and What We Can Learn from It. Does coaching raise grades when run at scale?”Oreopoulos and Petronijevic 2019, NBER Working Paper 26059. Download · Source
- Who was studied: About 25,000 students on three Canadian campuses over five years.
- How: RCTs of online, text-message and face-to-face coaching.
- What it found: No effect on grades or persistence; the trials rule out effects above 0.07 SD. Study time rose about two hours a week and well-being improved. Students studied 5-8 hours a week less than they planned, and coaching did not close that gap.
- Why it matters for us: One-on-ones can change habits without changing scores, so they should not be counted as the learning engine.
- Where we use it: Hybrid human-AI literature review (coaching moves decisions more than grades; design rule 3).
Why Do College Going Interventions Work? Does in-person mentoring help students at the margin of applying to college?
Section titled “Why Do College Going Interventions Work? Does in-person mentoring help students at the margin of applying to college?”Carrell and Sacerdote 2013, NBER Working Paper 19031. Download · Source
- Who was studied: Grade 12 students in New Hampshire, statewide across several schools, who were at the margin of applying to college.
- How: RCT of weekly in-person mentors, compared with information-only and cash-only arms.
- What it found: College-going among women rose 15 pp on an intent-to-treat basis (ITT: everyone offered the programme, whether or not they took it up) and about 30 pp among those treated. Information-only and cash-only arms had no effect. The authors read this as the mentor standing in for missing parent or teacher time.
- Why it matters for us: A human mentor helps where adult time is missing; information alone does not.
- Where we use it: Hybrid human-AI literature review (coaching evidence; gender caveat; what should stay human).
Using Goals to Motivate College Students: Theory and Evidence from Field Experiments. Should students commit to tasks or to scores?
Section titled “Using Goals to Motivate College Students: Theory and Evidence from Field Experiments. Should students commit to tasks or to scores?”Clark, Gill, Prowse and Rush 2017, NBER Working Paper 23638. Download · Source
- Who was studied: 3,971 students in a US college course.
- How: Two RCTs. Students set their own goals, either for a task (number of practice exams) or for performance (a target score).
- What it found: Task goals raised practice exams completed by about 0.1 SD and course performance by about 0.1 SD. Performance goals had small, non-significant effects. About a quarter of students hit a performance goal, against about half for task goals. Task goals helped men and did little for women.
- Why it matters for us: The weekly commitment should be a task the student controls (“finish 3 practice sets”), not a target score.
- Where we use it: Hybrid human-AI literature review (design rule 1; spec conflict on score targets).
Nudging at Scale: Experimental Evidence from FAFSA Completion Campaigns. Do mass reminder messages change what students do?
Section titled “Nudging at Scale: Experimental Evidence from FAFSA Completion Campaigns. Do mass reminder messages change what students do?”Bird et al. 2019, NBER Working Paper 26158. Download · Source
- Who was studied: Over 800,000 US students in national and state campaigns about applying for financial aid (FAFSA).
- How: Two RCTs varying the message framing, timing and offer of one-on-one advising.
- What it found: No effect on aid receipt or enrolment, and no variation worked. The authors suggest the messages were generic, one-way and from a sender students had little connection to.
- Why it matters for us: Generic reminders from the coach will not move students; contact has to be personal and backed by their teacher.
- Where we use it: Hybrid human-AI literature review (nudges; design rule 8; spec conflict on between-session contact).
How an Artificially Intelligent Virtual Assistant Helps Students Navigate the Road to College. Can a chatbot backed by human counsellors get admitted students to enrol?
Section titled “How an Artificially Intelligent Virtual Assistant Helps Students Navigate the Road to College. Can a chatbot backed by human counsellors get admitted students to enrol?”Page and Gehlbach 2017, AERA Open 3(4). Download · Source
- Who was studied: 7,489 students admitted to Georgia State University.
- How: RCT of an AI text-message chatbot that answered students’ own questions and passed unanswered ones to human counsellors.
- What it found: On-time enrolment rose 3.3 pp.
- Why it matters for us: When the coach cannot answer, the question should go to a teacher, and the teacher’s answer should be fed back into the coach.
- Where we use it: Hybrid human-AI literature review (nudges; design rule 9).
Getting students on track for graduation: Impacts of the Early Warning Intervention and Monitoring System after one year. Do early-warning flags cut absence and course failure?
Section titled “Getting students on track for graduation: Impacts of the Early Warning Intervention and Monitoring System after one year. Do early-warning flags cut absence and course failure?”Faria et al. 2017, IES Regional Educational Laboratory Midwest report (REL 2017-272). Download · Source
- Who was studied: 37,671 Grade 9-10 students in 73 US high schools.
- How: Cluster RCT (whole schools assigned by chance) of an early-warning system (EWIMS).
- What it found: Chronic absence fell from 14% to 10% and course failure from 26% to 21% in year one, though implementation was low in nearly all schools. No effect on credits earned, or on how often staff reviewed data or what they offered. The report says it is not clear which staff actions caused the gains.
- Why it matters for us: Flags to teachers help a little even when used badly, but each flag needs a reason and a suggested action, and we should track whether teachers act.
- Where we use it: Hybrid human-AI literature review (early-warning flags; design rule 6).
Step-based tutoring and tutoring software
Section titled “Step-based tutoring and tutoring software”The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Does checking each step of a solution teach more than marking the final answer?
Section titled “The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Does checking each step of a solution teach more than marking the final answer?”VanLehn 2011, Educational Psychologist 46(4). Download · Source
- Who was studied: Experiments that compared two kinds of tutoring on the same STEM content, mostly with US college students and many short lab studies (28 step-based versus no-tutoring comparisons).
- How: Review of pairwise experiments, effects in d (a standardised mean difference, read like SD).
- What it found: Against no tutoring: step-based tutors d = 0.76, human tutors 0.79, substep (dialogue) tutors 0.40, answer-only systems 0.31. Substep beat step-based by only 0.16, and human tutors beat step-based by 0.21. The 0.31 is imported from an older review of different studies; the two direct step-versus-answer comparisons give 0.40.
- Why it matters for us: The right grain for the coach is one line of written working checked as it is written; finer dialogue adds little.
- Where we use it: Meta-analysis section 3.11; Step-based tutoring literature review (lead finding, design rules 1-3); Adaptive platforms literature review (design rule 5).
The Andes physics tutoring system: Lessons learned. Does step-by-step physics homework feedback raise exam scores?
Section titled “The Andes physics tutoring system: Lessons learned. Does step-by-step physics homework feedback raise exam scores?”VanLehn et al. 2005, International Journal of Artificial Intelligence in Education 15(3). Download · Source
- Who was studied: Introductory physics students at the US Naval Academy over five fall semesters, 1999-2003: 628 Andes and 438 control students on hour exams, and 931 students on the 2003 final.
- How: Quasi-experiment. Andes sections did homework in the tutor and control sections did similar homework on paper, with the same textbook and lectures. Sections were not randomised and the Andes instructors were co-authors.
- What it found: On show-your-work hour exams, d ranged from 0.21 to 0.92 by year (pooled 0.61). On the answer-only multiple-choice final it was 0.25 overall, not significant for engineering (0.22) or science majors (0.18), and 0.5 for other majors. Andes improved drawings (1.21) and variable definitions (0.69) but barely equations (0.11) or final answers (-0.08).
- Why it matters for us: JEE scores only the answer, so expect gains well below show-your-work results, and smallest for strong science students.
- Where we use it: Meta-analysis section 3.11; Step-based tutoring literature review (what step feedback improves; design rules 2, 5 and 11).
Intelligent tutoring systems and learning outcomes: a meta-analysis. How do tutoring systems compare with teachers and textbooks?
Section titled “Intelligent tutoring systems and learning outcomes: a meta-analysis. How do tutoring systems compare with teachers and textbooks?”Ma, Adesope, Nesbit and Liu 2014, Journal of Educational Psychology 106(4). Download · Source
- Who was studied: 14,321 learners of all ages and subjects, in 107 effect sizes.
- How: Meta-analysis (a statistical pooling of many studies’ results), random effects, with effects in g (a standardised mean difference corrected for small samples).
- What it found: g = 0.41 overall. Against large-group teaching 0.42, against textbooks 0.35, against human tutoring -0.11 (no difference) and against small-group teaching 0.05. High school 0.40 (14 effect sizes); physics 0.38 (24); chemistry 0.16 (2).
- Why it matters for us: A realistic tutoring-software effect for secondary physics is about 0.4 SD against lectures, and none against a teacher with a small group.
- Where we use it: Adaptive platforms literature review (effect sizes); Step-based tutoring literature review (field effects are smaller).
A meta-analysis of the effectiveness of intelligent tutoring systems on K-12 students’ mathematical learning. Do maths tutoring systems work in ordinary US schools?
Section titled “A meta-analysis of the effectiveness of intelligent tutoring systems on K-12 students’ mathematical learning. Do maths tutoring systems work in ordinary US schools?”Steenbergen-Hu and Cooper 2013, Journal of Educational Psychology 105(4). Download · Source
- Who was studied: K-12 maths students, mostly in the US, in 34 samples from 26 reports (1997-2010), many of them Cognitive Tutor field trials.
- How: Meta-analysis, mostly against regular classroom teaching.
- What it found: Mean g from 0.01 to 0.09 depending on the model, larger for programmes shorter than a school year. Low achievers did worse: -0.42 (2 effect sizes, unadjusted) or -0.18 (3, adjusted) in a fixed-effect model, not significant under random effects.
- Why it matters for us: In real school use step-based maths tutors averaged close to zero, so supervision and fit matter more than the software, especially for weaker students.
- Where we use it: Step-based tutoring literature review (field effects; who benefits; design rule 12).
Do intelligent tutoring systems benefit K-12 students? A meta-analysis and evaluation of heterogeneity of treatment effects in the U.S. What does tutoring software achieve in US schools under strict study standards?
Section titled “Do intelligent tutoring systems benefit K-12 students? A meta-analysis and evaluation of heterogeneity of treatment effects in the U.S. What does tutoring software achieve in US schools under strict study standards?”Leite et al. 2025, arXiv preprint. Download · Source
- Who was studied: US K-12 students in maths and reading: 18 studies, 77 effect sizes.
- How: Meta-analysis restricted to studies meeting What Works Clearinghouse-like standards.
- What it found: g = 0.27 overall; 0.20 on standardised tests against 0.483 (not significant) on researcher-developed tests. Effects were lower in rural schools. Worked examples, duration and immediate measurement were the top moderators.
- Why it matters for us: About 0.2 SD on independent tests is the realistic size for tutoring software in schools, well below lab results.
- Where we use it: Meta-analysis section 3 (planning ranges table, step-based tutoring software at scale) and section 3.11; Adaptive platforms literature review (design rules 5 and 12).
The role of mastery learning in an intelligent tutoring system: principal stratification on a latent variable. Did students who moved on through mastery gain more from Cognitive Tutor?
Section titled “The role of mastery learning in an intelligent tutoring system: principal stratification on a latent variable. Did students who moved on through mastery gain more from Cognitive Tutor?”Sales and Pane 2017 (revised 2018), arXiv preprint. Download · Source
- Who was studied: High- and middle-school algebra students in the Cognitive Tutor Algebra I trial: 147 schools in seven states, about 18,700 high-school students.
- How: Re-analysis of a cluster RCT, using principal stratification (a method that estimates effects for students grouped by how they would have used the tutor).
- What it found: The tutor may be more effective for students who were promoted through mastery more often, but the authors cannot say why. The paper restates the trial’s main effects: high school -0.10 SD in year 1 and 0.22 SD in year 2; middle school -0.03 and 0.19.
- Why it matters for us: Expect little in the first cycle of a new tool, and do not assume mastery gating alone drives the gain.
- Where we use it: Adaptive platforms literature review (mastery gating; design rules 8 and 11; spec conflict on a 10-week pilot).
Mastery learning in practice: a (mostly) descriptive analysis of log data from the Cognitive Tutor Algebra I effectiveness trial. How did students and teachers actually use mastery gating?
Section titled “Mastery learning in practice: a (mostly) descriptive analysis of log data from the Cognitive Tutor Algebra I effectiveness trial. How did students and teachers actually use mastery gating?”Israni, Sales and Pane 2018, arXiv preprint. Download · Source
- Who was studied: 4,460 students in the logs of the Cognitive Tutor Algebra I trial.
- How: Descriptive and observational analysis of software logs.
- What it found: Median use was 33.5 hours in year 1 and 23.8 in year 2. Students often moved on without mastery and worked units out of order, more so in year 2 when the effect was larger. Teacher reassignment before mastery was associated with lower post-test scores, varying across classrooms.
- Why it matters for us: Use mastery thresholds, but let a teacher move a student on and log it, and expect usage to fall over time.
- Where we use it: Adaptive platforms literature review (usage falls short of plan; mastery gating; design rule 8).
How to build the steps: examples, subgoals and self-explanation
Section titled “How to build the steps: examples, subgoals and self-explanation”From example study to problem solving: Smooth transitions help learning. Does removing worked steps one at a time help students move from examples to problems?
Section titled “From example study to problem solving: Smooth transitions help learning. Does removing worked steps one at a time help students move from examples to problems?”Renkl, Atkinson and Maier 2000, University of Freiburg Research Report 140. Download · Source
- Who was studied: Experiment 1: two Grade 9 classes (20 and 15 students) at a lowest-track German school, in one physics lesson on electricity. Experiment 2: 54 US psychology undergraduates learning probability.
- How: Experiment 1 was a quasi-experiment comparing backward fading (the last step left blank first, then more) with example-problem pairs. Experiment 2 was a lab RCT of forward fading.
- What it found: Experiment 1: near transfer 79.4% against 62.2% (significant; g ≈ 0.64, 95% CI -0.05 to 1.32); far transfer 36.3% against 21.1% (not significant). Experiment 2: near transfer 53.9% against 43.8% (significant; g ≈ 0.29); far transfer 38.7% against 43.4% (not significant). Fading reduced errors during learning.
- Why it matters for us: Fading worked steps is a cheap bridge from examples to practice, but its gain is shown only for near transfer in two small studies.
- Where we use it: Meta-analysis section 3.3 and section 3 planning ranges table (faded worked examples); What works section b; Step-based tutoring literature review (design rule 6).
The subgoal learning model: Creating better examples so that students can solve novel problems. Does naming the purpose of a group of steps help students solve changed problems?
Section titled “The subgoal learning model: Creating better examples so that students can solve novel problems. Does naming the purpose of a group of steps help students solve changed problems?”Catrambone 1998, Journal of Experimental Psychology: General 127(4). Download · Source
- Who was studied: US college students learning Poisson probability from worked examples: 150 (Experiment 1), 180 (Experiment 2) and 90 (Experiment 4).
- How: Four lab experiments, comparing examples with and without a label naming each group of steps (a subgoal).
- What it found: In Experiment 2 (30 per group), novel problems solved out of 4: no label 2.33 at once and 1.77 a week later; surface label 3.47 and 1.80; abstract label 3.17 and 3.00. Students with weaker maths needed labels tied to the problem’s features.
- Why it matters for us: Each rung of a coach lesson should carry a named purpose, abstract for strong students and tied to the problem for weaker ones.
- Where we use it: Step-based tutoring literature review (subgoals; design rule 4).
Training learners to self-explain: Designing instructions and examples to improve problem solving. Do given or self-written subgoal labels work better?
Section titled “Training learners to self-explain: Designing instructions and examples to improve problem solving. Do given or self-written subgoal labels work better?”Margulieux, Morrison, Guzdial and Catrambone 2016, ICLS 2016 conference paper. Download · Source
- Who was studied: US introductory programming students, novices only; 120 analysed of 175.
- How: Lab RCT of worked-example and practice pairs with no labels, given subgoal labels, or labels students wrote themselves after training.
- What it found: Given labels added 7.8 points and self-written labels 8.6 points out of 44 against no labels (partial omega squared .08). The two worked equally well on average; writing your own needs training first.
- Why it matters for us: The coach can supply subgoal labels; asking students to write their own only pays after training.
- Where we use it: Step-based tutoring literature review (subgoals; design rule 4).
Self-explanations: How students study and use examples in learning to solve problems. What do strong students do differently with worked examples?
Section titled “Self-explanations: How students study and use examples in learning to solve problems. What do strong students do differently with worked examples?”Chi, Bassok, Lewis, Reimann and Glaser 1989, Cognitive Science 13. Download · Source
- Who was studied: Students with no college physics studying worked mechanics examples: 10 recruited, 8 analysed (4 “Good”, 4 “Poor”).
- How: Think-aloud study. Correlational, not an experiment.
- What it found: Good students solved 82% of problems and Poor students 46%. Good students produced 15.3 physics explanations per example against 2.8.
- Why it matters for us: Strong students explain why each step is there, which is the habit the coach’s “why” prompts try to build; this tiny study shows a pattern, not a cause.
- Where we use it: Step-based tutoring literature review (self-explanation at the steps).
Inducing self-explanation: a meta-analysis. Does prompting students to explain to themselves improve learning?
Section titled “Inducing self-explanation: a meta-analysis. Does prompting students to explain to themselves improve learning?”Bisra, Liu, Nesbit, Salimi and Winne 2018, Educational Psychology Review 30. Download · Source
- Who was studied: 5,917 participants in 64 reports (69 effect sizes), from primary school to professionals, mostly in North America and Europe; 1,415 in high school.
- How: Meta-analysis of experiments that prompted learners to self-explain while studying or solving.
- What it found: g = 0.55 overall; 0.67 against no extra explanation and 0.35 against being given the explanation; 0.43 in high school; 0.41 with time held equal; 0.36 while studying worked examples and 0.48 while solving problems. Content-general prompts 0.68 against specific 0.51 (difference not significant). Multiple-choice prompts 0.24 (not significant, 2 effect sizes); fill-in-the-blank 0.90 (2 effect sizes). Publication bias was checked only by fail-safe N, so the means are upper bounds.
- Why it matters for us: One or two short “why does this step follow?” prompts at the key principle are worth their practice time.
- Where we use it: Meta-analysis section 3.5 and section 3 planning ranges table (self-explanation prompts), and 3.11 (publication bias); Step-based tutoring literature review (design rule 8).
The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. What is the unit a lesson step should practise?
Section titled “The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. What is the unit a lesson step should practise?”Koedinger, Corbett and Perfetti 2012, Cognitive Science 36(5). Download · Source
- Who was studied: Not an effect study; built on many classroom experiments at the Pittsburgh Science of Learning Center.
- How: Theoretical framework.
- What it found: Defines a knowledge component as a unit of knowledge inferred from performance on related tasks, typically a step of about ten seconds, such as applying one theorem. It separates memory and fluency, induction and refinement, and understanding and sense-making, and treats combining two steps as its own piece of knowledge.
- Why it matters for us: Each coach lesson should tag steps by the knowledge they practise, and must end with the step that joins the rungs, practised unaided.
- Where we use it: Step-based tutoring literature review (what a step is; design rule 9).
An astonishing regularity in student learning rate. How many practice opportunities does it take to learn a skill?
Section titled “An astonishing regularity in student learning rate. How many practice opportunities does it take to learn a skill?”Koedinger, Carvalho, Liu and McLaughlin 2023, PNAS 120(13). Download · Source
- Who was studied: 6,946 learners from late elementary school to college in 27 datasets (1.3 million observations) from tutors, games and online courses in maths, science and language.
- How: Learning-curve models fitted to the logs.
- What it found: Initial accuracy was about 55% in the lower half of students and 75% in the upper half. Learning rate was about 0.1 log odds, or 2.5 percentage points of accuracy, per practice opportunity, similar across students. It took about seven opportunities per skill to reach 80%. A 2026 preprint questions the regularity; it was not opened.
- Why it matters for us: One lesson after a wrong answer is one or two opportunities, so the coach must queue about seven spaced practice items per skill.
- Where we use it: Step-based tutoring literature review (how much practice one lesson buys; design rule 10).
The four-component instructional design model (4C/ID): A meta-analysis on use and effect. Do whole-task programmes with fading support work?
Section titled “The four-component instructional design model (4C/ID): A meta-analysis on use and effect. Do whole-task programmes with fading support work?”Costa, Miranda and Melo 2022, Learning Environments Research 25(2). Download · Source
- Who was studied: 12 studies with samples of 22 to 160.
- How: Meta-analysis of programmes built on the 4C/ID model, which sequences whole tasks from simple to complex and fades support within each level.
- What it found: d = 0.79 (95% CI 0.50 to 1.08), with heavy heterogeneity (I squared 78.6%, meaning results varied a lot between studies). Effects were larger in higher education.
- Why it matters for us: Weak support for building lessons around whole problems with fading support; too few, mixed studies to size an effect.
- Where we use it: Step-based tutoring literature review (whole tasks; weak-evidence list).
Synthesis of discipline-based education research in physics. What does physics education research say about teaching problem solving?
Section titled “Synthesis of discipline-based education research in physics. What does physics education research say about teaching problem solving?”Docktor and Mestre 2014, Physical Review Special Topics - Physics Education Research 10. Download · Source
- Who was studied: Hundreds of physics education studies, mostly US.
- How: Research synthesis for the US National Research Council.
- What it found: Explicit problem-solving frameworks, modelled and reinforced through a course, give higher course performance. Context-rich problems help when used with a strategy and groups. Students find it hard to diagnose their own errors: one third to one half recognised using the wrong principle. Web homework against paper gave mixed results.
- Why it matters for us: The coach should use one fixed sequence for every physics problem (describe, plan, execute, check), and not rely on students finding their own errors from a solution sheet.
- Where we use it: Step-based tutoring literature review (whole tasks and physics practice).
Instructional strategies that foster effective problem-solving. What physics teaching strategies build problem-solving skill?
Section titled “Instructional strategies that foster effective problem-solving. What physics teaching strategies build problem-solving skill?”Singh, Maries, Heller and Heller 2023, arXiv book chapter. Download · Source
- Who was studied: Not an effect study; a narrative review of physics problem-solving instruction, including the Minnesota cooperative-group work.
- How: Narrative review by physics education researchers.
- What it found: Supports a four-step framework, outlining before solving and fading of scaffolds. It summarises Yerushalmi et al. 2012: students who could regain 50% of lost points by correcting a test did better using their own resources than when given a detailed solution.
- Why it matters for us: Test review in the coach should come with a reason to do it and little scaffolding, since a full solution can make reflection shallow.
- Where we use it: Step-based tutoring literature review (whole tasks and physics practice; design rule 14).
Adaptive learning platforms
Section titled “Adaptive learning platforms”Disrupting education? Experimental evidence on technology-aided instruction in India. Does adaptive software with an instructor raise learning in Delhi?
Section titled “Disrupting education? Experimental evidence on technology-aided instruction in India. Does adaptive software with an instructor raise learning in Delhi?”Muralidharan, Singh and Ganimian 2017, NBER Working Paper 22923 (published in the American Economic Review 109(4), 2019). Download · Source
- Who was studied: 619 Grade 6-9 students from Delhi public schools at after-school Mindspark centres. The average Grade 6 student was about 2.5 grade levels behind, and 4.5 by Grade 9.
- How: Lottery RCT. 45 minutes of adaptive software plus 45 minutes of group instruction, six days a week, for 4.5 months.
- What it found: Intent-to-treat effects of 0.36 SD in maths and 0.22 SD in Hindi. The instrumental-variable estimate (the effect per amount actually attended) for 90 days of attendance was 0.59 SD in maths and 0.36 in Hindi. Mean attendance was 58%. Maths gains were on below-grade-level questions only, with no significant gain on grade-level school exams. The published version is quoted at 0.37 and 0.23 SD; that version was not opened.
- Why it matters for us: Level-targeted practice with an adult in the room works in India, but gains appear where the content is pitched, so a coach pitched below JEE level will not show up in JEE scores; the students were also much younger and further behind than ours.
- Where we use it: Meta-analysis section 3.11 and Appendix B; Adaptive platforms, India context, Learning science and Medium (poorer countries) literature reviews.
Teaching teachers to use computer assisted learning effectively: experimental and quasi-experimental evidence. Does coaching teachers to use Khan Academy raise maths scores?
Section titled “Teaching teachers to use computer assisted learning effectively: experimental and quasi-experimental evidence. Does coaching teachers to use Khan Academy raise maths scores?”Oreopoulos, Gibbs, Jensen and Price 2024, NBER Working Paper 32388. Download · Source
- Who was studied: US elementary and middle-school students; about 11,000 according to Khan Academy’s summary.
- How: Two field experiments with weekly coaching for teachers on mastery learning with Khan Academy.
- What it found: Intent-to-treat effects of 0.12 to 0.22 SD in maths. Gains concentrated in classrooms averaging 35 or more minutes a week; teacher buy-in and a fixed practice slot predicted dosage.
- Why it matters for us: Aim for at least 45 minutes a week of actual problem practice, protected by a teacher who owns the slot.
- Where we use it: Adaptive platforms literature review (dose; design rule 3).
Estimating the causal effects of Khan Academy MAP Accelerator across demographic subgroups. How much does using Khan Academy’s MAP Accelerator relate to maths growth?
Section titled “Estimating the causal effects of Khan Academy MAP Accelerator across demographic subgroups. How much does using Khan Academy’s MAP Accelerator relate to maths growth?”Grimaldi, Weatherholtz and Hill 2022, EDM 2022 conference paper (industry track). Download · Source
- Who was studied: 181,359 US Grade 3-8 students in 2020-21.
- How: Observational, difference-in-differences (comparing changes over time between heavier and lighter users).
- What it found: 0.26 SD for students using it 30 or more minutes a week against under 15 minutes. Only 5% reached the recommended dose and 41% never used it. Usage was mostly predicted by district, school and teacher factors.
- Why it matters for us: Most students do not use a platform at the planned dose unless the school makes them, so we should budget for low realised use.
- Where we use it: Adaptive platforms literature review (usage falls short of plan; design rule 2).
Khan Academy efficacy results, November 2024. How much growth do students who use Khan Academy at the recommended dose show?
Section titled “Khan Academy efficacy results, November 2024. How much growth do students who use Khan Academy at the recommended dose show?”Yamkovenko and Bergstrom 2024, Khan Academy blog (company-reported). Download · Source
- Who was studied: About 350,000 US Grade 3-8 students in 2022-23.
- How: Company-reported observational study, not independent.
- What it found: About 20% greater-than-expected growth on the NWEA MAP test (a common US adaptive test) for students with 18 or more hours a year. Only about 9% reached that dose. Khan’s mastery levels correlate with gains in its own data.
- Why it matters for us: Even the provider’s own figures show fewer than one in ten students reaching the dose, which supports a protected, supervised slot.
- Where we use it: Adaptive platforms literature review (usage; mastery gating; design rule 2).
Research on the use of Khan Academy in schools. How did nine California schools use Khan Academy, and what went with high use?
Section titled “Research on the use of Khan Academy in schools. How did nine California schools use Khan Academy, and what went with high use?”Murphy, Gallagher, Krumm, Mislevy and Hafter 2014, SRI Education report. Download · Source: no public link recorded (archived from Avanti’s nvs-lakshya folder)
- Who was studied: 1,694 to 2,246 students a year at 9 California sites.
- How: Implementation study with a correlational outcome analysis.
- What it found: Positive associations only; the authors say they are not causal. The site with the highest use had one-to-one devices, mandated goals with consequences, close teacher monitoring and 90-minute maths blocks.
- Why it matters for us: High use came from adults setting goals and watching progress, not from the software alone.
- Where we use it: Adaptive platforms literature review (usage; design rule 1).
Education technology: an evidence-based review. Which kinds of education technology have been shown to work in experiments?
Section titled “Education technology: an evidence-based review. Which kinds of education technology have been shown to work in experiments?”Escueta, Quan, Nickow and Oreopoulos 2017, NBER Working Paper 23744 (published in the Journal of Economic Literature in 2020). Download · Source
- Who was studied: Experimental studies of education technology, mostly from high-income countries and higher education.
- How: Review of RCTs, including 29 of computer-assisted learning (CAL).
- What it found: 20 of 29 CAL trials were positive, 15 of those in maths. Access to devices alone does little. Fully online courses did slightly worse than face-to-face; blended courses were not significantly different.
- Why it matters for us: Practice software is the most promising category, and it should add to a teacher’s work, not replace it.
- Where we use it: Adaptive platforms literature review (evidence table); Medium (poorer countries) and Medium (richer countries) literature reviews (online against face-to-face).
Educational technology in developing countries: a review of the evidence. What does causal research on education technology in poorer countries show?
Section titled “Educational technology in developing countries: a review of the evidence. What does causal research on education technology in poorer countries show?”Rodriguez-Segura 2021, author draft of the World Bank Research Observer 37(2) paper (2022). Download · Source
- Who was studied: 81 core studies of education technology with causal identification in developing countries.
- How: Review.
- What it found: Self-led learning software had a median effect of 0.29 SD. One-laptop-per-child evaluations found null learning effects. Dosage, adaptivity and incentives are under-tested; the one study that doubled the dose found no further gain, and in one Chinese study the ed-tech arm did no better than extra paper practice (both reported second-hand).
- Why it matters for us: Devices alone do nothing; the software and the routine around it carry the effect.
- Where we use it: Adaptive platforms literature review (dose; weak-evidence list; design rule 9); India context literature review; Medium (poorer countries) literature review (published version).
Performance comparison of an AI-based adaptive learning system in China. Did Squirrel AI beat expert teachers in a short trial?
Section titled “Performance comparison of an AI-based adaptive learning system in China. Did Squirrel AI beat expert teachers in a short trial?”Cui, Xue and Thai 2019, arXiv paper (company authors). Download · Source
- Who was studied: 203 Grade 8 maths students in China; 163 finished (90 of 101 treatment, 73 of 102 control).
- How: Short RCT over three days of a school vacation, written by company authors.
- What it found: The Squirrel AI group scored higher than the expert-teacher group. Attrition was 11% against 28%.
- Why it matters for us: Not credible evidence for adaptive AI at our level: three days, company-written and uneven drop-out.
- Where we use it: Adaptive platforms literature review (Squirrel AI, Brilliant, Alcumus).
Assessing the impact of Brilliant.org on enhancing mathematics academic performance among high school students in Colombia: a quasi-experimental study. Has Brilliant.org been evaluated?
Section titled “Assessing the impact of Brilliant.org on enhancing mathematics academic performance among high school students in Colombia: a quasi-experimental study. Has Brilliant.org been evaluated?”de la Puente and Perez 2023, Mathematics Teaching Research Journal 15(2) (study protocol). Download · Source
- Who was studied: A planned 60-student quasi-experiment with high-school students in Colombia.
- How: Study protocol only.
- What it found: No results are reported. No outcome evaluation of Brilliant was found.
- Why it matters for us: Brilliant has no evidence on learning, so it should not be built into the coach on its reputation.
- Where we use it: Adaptive platforms literature review (Squirrel AI, Brilliant, Alcumus; spec conflict on the multi-platform model).
Evaluation of Beast Academy in Mankato Area Public Schools. Does AoPS’s Beast Academy raise maths results?
Section titled “Evaluation of Beast Academy in Mankato Area Public Schools. Does AoPS’s Beast Academy raise maths results?”Gibney and Bojorquez 2022, WestEd technical report commissioned by Art of Problem Solving. Download · Source
- Who was studied: Grade 3-6 students in one US district (Mankato).
- How: Matched quasi-experiment, commissioned by the provider.
- What it found: Positive and significant after two years.
- Why it matters for us: It is a different AoPS product for younger children, so it is not evidence for Alcumus.
- Where we use it: Adaptive platforms literature review (Squirrel AI, Brilliant, Alcumus).
Student Growth Summary Report, Spring 2024-2025 (NWEA MAP Growth). What do Alpha School’s own test reports show?
Section titled “Student Growth Summary Report, Spring 2024-2025 (NWEA MAP Growth). What do Alpha School’s own test reports show?”Alpha School 2025, self-generated NWEA MAP report (provider data, not an evaluation). Download · Source
- Who was studied: Alpha School students from kindergarten to Grade 8: 148 maths growth events, none for Grades 9-10.
- How: Growth report on the NWEA MAP test, generated by Alpha from its own testing, with no comparison group.
- What it found: Autumn maths percentiles were already 97-99 in eight of nine grades (82 in Grade 5). Observed growth was 1.3 to 2.2 times NWEA’s projected growth in maths and 1.1 to 1.9 times in reading, depending on grade (the strand author’s arithmetic from the published table).
- Why it matters for us: Alpha’s numbers come from already high-scoring students without a comparison group, so they are not a basis for copying its multi-platform model.
- Where we use it: Adaptive platforms literature review (Alpha School claimed against documented; spec conflict on the Alpha-inspired model).