K-12Issue 104 records
K–12
Fresh K-12 signal centers on measured tutoring experiments and state procurement guardrails. A two-year Tennessee cluster RCT finds Khanmigo access yields modest math gains similar to non-AI practice while sparse student dialogue makes engagement the binding constraint; a companion NUMI field experiment shows LLM help adds value mainly when embedded in mastery workflows that force productive recovery from mistakes. A September 13 UK GCSE science micro-RCT reports a preliminary positive ITT effect with high attrition and short follow-up, arguing schools need rapid cumulative randomised evaluation as products drift. Idaho’s GenAI education framework, tied to 2026 legislation, requires vendor DPAs and multi-statute privacy vetting before platform adoption. Together the sources caution districts against equating tool access or feature claims with learning impact without engagement design, structured practice, privacy contracts, and evaluation that keeps pace with version change.
- Evidence records
- 4
- Cross-source patterns
- 2
- Evidence classes
- 3 academic research1 standards or public-body guidance
- Outcomes
- 3 mixed1 emerging
- Source freshness
- 3 recent1 new this fortnight
- Research completed
- 2026-09-15
Choose a role to see its takeaway beside every record in the ledger.
Patterns across the evidence
Synthesis · Lighthouse Advisory interpretation
AI tutoring gains hinge on engagement design and mastery structure, not access alone
Across the Tennessee Khanmigo cluster RCT and the NUMI mastery experiment, measured benefits are bounded by whether students actually enter productive dialogue or post-error scaffolds. Access and AI features without engagement routines or mastery workflows tend to reproduce ordinary practice-software effects or dilute delayed learning signals.
Operating questionWhat classroom and platform structures will force productive tutor use after mistakes before we scale AI tutoring licenses?
Supporting evidenceAnnenberg Institute at Brown University / NBER (Oreopoulos & Low)Annenberg Institute at Brown University (Oreopoulos, Liut, Sungu & Low)
Districts need evaluation and contracting rhythms that keep pace with drifting AI products
The Medly micro-RCT argues conventional trials lag version change, while Idaho’s framework demands DPAs and recurring vendor vetting. Together they imply SLED buyers should couple rapid cumulative evaluation with contractual privacy and version discipline rather than one-time adoption decisions.
Operating questionHow will we version-freeze, re-evaluate, and re-contract AI school tools as models and statutes change?
Supporting evidenceWhat Worked Education / Durham University (Harrison et al.)Idaho Department of Education
Evidence ledger
Full record · every source keeps its link and limitations
Tennessee cluster RCT finds Khanmigo access gains resemble non-AI practice as engagement stays sparse
In a two-year cluster randomized trial across 18 Tennessee middle schools, assigning Khan Academy with Khanmigo (configured to coach rather than give answers) during daily remedial math raised achievement by about 1.3 national percentile ranks per term (roughly 0.06–0.08 SD over a school year), with an implied ~0.14 SD for a full year of active participation. Those gains resemble Khan Academy practice without AI assistance. Nearly all students tried the tutor once, but the median student messaged it on only a third of practice days and in only 17% of mistake sessions, usually with bare answers or suggested-prompt clicks—so engagement, not access alone, appears binding.
Why it matters, evidence and limitations
- Why it matters
- Directly relevant to U.S. district decisions on AI tutoring inside RTI/intervention blocks: procurement and pilot design should measure dialogue quality and sustained use, not license counts. Transferable caution for other SLED tutoring pilots claiming personalized AI coaches.
- Evidence and measured results
- Cluster RCT; AEA RCT Registry AEARCTR-0013519; University of Toronto REB. Intent-to-treat estimates from MAP/percentile outcomes; platform messaging logs document sparse substantive dialogue. Authors note gains resemble non-AI Khan Academy practice.
- Limitations and uncertainty
- Working paper; single Tennessee district; control condition may include other edtech; ITT averages over low AI use. Do not treat 0.14 SD active-participation implication as guaranteed for every rostered student.
NUMI field experiment ties LLM tutoring value to mastery workflows that recover from mistakes
A randomized field experiment with more than 6,000 middle-school students on the NUMI CAL platform crossed AI versus CAL-only support with mastery versus non-mastery progression. AI students progressed more slowly and attempted fewer questions but were more accurate when they attempted; the clearest mechanism was better next-attempt correctness and fewer attempts to recover after mistakes, with more time on structured support. Mastery raised three-correct-in-a-row attainment but did not alone improve delayed learning. The most encouraging delayed-test signal appeared when AI was embedded in the mastery workflow, with marginally significant gains concentrated on practiced material—supporting the claim that LLM tutoring adds value over standard CAL only with structure that turns mistakes into productive learning.
Why it matters, evidence and limitations
- Why it matters
- Helps districts interpret mixed AI tutoring pilots: product selection and classroom workflow must enforce productive struggle and mastery gates, not open-ended chat. Relevant to instructional design RFPs and professional learning for AI-supported practice.
- Evidence and measured results
- Registered field experiment (AEA RCT Registry AEARCTR-0018678) with delayed assessment about one week later covering practiced and unpracticed items; factorial randomization across AI, mastery, and topics.
- Limitations and uncertainty
- Delayed outcomes short; encouraging AI+mastery results are marginally significant and concentrated on practiced Exercise 1 content. Research platform (NUMI) may differ from commercial suites. Same district partnership as the Khanmigo RCT.
UK GCSE science micro-RCT reports preliminary AI tutoring gains while highlighting evaluation lag and attrition
A four-week multisite individually randomised evaluation of Medly, an AI tutoring platform, in GCSE science found primary ITT Hedges’ g = 0.33 (95% CI 0.18–0.48) versus business-as-usual self-directed revision among students with post-tests (644 of 929 baseline). Positive subject estimates and no clear Pupil Premium differential are reported, but authors label results preliminary because attrition was 30.7%, outcomes were curriculum-aligned rather than standardised, follow-up was short, and process evaluation response was limited. They argue educational AI needs rapid, repeatable micro-RCTs because interventions drift while conventional trials lag.
Why it matters, evidence and limitations
- Why it matters
- Transferable scrutiny for U.S. districts: even positive AI tutoring estimates can be fragile under attrition and short horizons, and product version drift can obsolete slow evaluations. Supports building cumulative local randomised tests into edtech adoption—not only one-off pilots.
- Evidence and measured results
- Individually randomised multisite trial; ITT with Hedges’ g; baseline n=929; post-test n=644; subject-specific estimates; Pupil Premium interaction examined. Limitations section emphasizes attrition imbalance and short-term topic learning.
- Limitations and uncertainty
- UK GCSE context; substantial attrition (higher in control); non-standardised curriculum-aligned tests; four-week horizon; arXiv preprint; exploratory engagement analyses are not causal.
Idaho GenAI education framework requires DPAs and multi-statute privacy vetting for school platforms
Following 2026 Senate Bill 1227, the Idaho Department of Education published a statewide Generative AI in Education Framework covering literacy, age appropriateness, appropriate/inappropriate uses, academic integrity, and platform purchasing. On purchasing, it instructs schools to vet vendors on data use, whether data are used for model training, retention/access/deletion, safety controls, and compliance with COPPA, FERPA, PPRA, CIPA, IDEA, Title VI, state student privacy laws, and state AI legislation, and states that after selection a district, school, or charter should always require a Data Privacy Agreement covering ownership, use limits, data types, compliance, security, and retention/deletion. The Department commits to at least annual review.
Why it matters, evidence and limitations
- Why it matters
- Actionable SEA template for U.S. district AI procurement and staff/student use policy. Newly relevant as states translate 2026 AI legislation into operable K-12 purchasing controls beyond generic ethics statements.
- Evidence and measured results
- Official state education agency framework PDF with platform-purchasing checklist and DPA component list; legislative anchor SB 1227; includes draft policy annex materials.
- Limitations and uncertainty
- Guidance and draft policy language, not an audit of district compliance or measured learning outcomes. Idaho statute context; other states may differ. Annual revision means controls may change.
How to read this edition
Source findings, measured results and limitations come from the cited publications. Patterns, operating questions, role takeaways and implementation considerations are Lighthouse Advisory interpretation, stated as questions to validate locally rather than guaranteed outcomes. Vendor and operator claims are labeled as claims. Full research method.
- Academic research
- Research produced through an academic institution or peer-reviewed venue.
- Standards or public-body guidance
- Normative or advisory guidance from a standards body or public institution.