From the K–12 edition of September 15, 2026
NUMI field experiment ties LLM tutoring value to mastery workflows that recover from mistakes
Annenberg Institute at Brown University (Oreopoulos, Liut, Sungu & Low) · K-12 public education / education research · United States — Hamilton County Schools, Tennessee middle schools
- Publisher
- EdWorkingPapers
- Original publication
- August 17, 2026 (EdWorkingPaper 26-1552)
- Source retrieved
- 2026-09-15
What happened
A randomized field experiment with more than 6,000 middle-school students on the NUMI CAL platform crossed AI versus CAL-only support with mastery versus non-mastery progression. AI students progressed more slowly and attempted fewer questions but were more accurate when they attempted; the clearest mechanism was better next-attempt correctness and fewer attempts to recover after mistakes, with more time on structured support. Mastery raised three-correct-in-a-row attainment but did not alone improve delayed learning. The most encouraging delayed-test signal appeared when AI was embedded in the mastery workflow, with marginally significant gains concentrated on practiced material—supporting the claim that LLM tutoring adds value over standard CAL only with structure that turns mistakes into productive learning.
Why it matters
Helps districts interpret mixed AI tutoring pilots: product selection and classroom workflow must enforce productive struggle and mastery gates, not open-ended chat. Relevant to instructional design RFPs and professional learning for AI-supported practice.
Evidence and measured results
Registered field experiment (AEA RCT Registry AEARCTR-0018678) with delayed assessment about one week later covering practiced and unpracticed items; factorial randomization across AI, mastery, and topics.
Limitations and uncertainty
Delayed outcomes short; encouraging AI+mastery results are marginally significant and concentrated on practiced Exercise 1 content. Research platform (NUMI) may differ from commercial suites. Same district partnership as the Khanmigo RCT.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-15; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Districts see uneven AI tutoring results because tools are bolted on without mastery structure that forces productive recovery from errors.
- Stakeholders
- Curriculum & instruction
- Math department chairs
- Edtech platform owner
- Research/assessment
- Discovery questions
- Can your practice platform enforce mastery gates and post-error scaffolds today?
- Do students currently use AI to skip struggle rather than repair misconceptions?
- What delayed-practice assessment will you run one week after a unit?
- Value hypothesis
- LLM support can beat standard CAL when embedded in mastery workflows that increase productive time after mistakes—without guaranteeing broad transfer gains.
- Potential engagement
- Short factorial classroom pilot comparing AI+mastery versus AI without mastery on one math topic with a delayed practiced-item check.
- Unsupported claims
- Unrestricted AI chat improves delayed learning by itself
- Mastery badges alone guarantee learning gains
- Guaranteed acceleration of pacing for all students
Pre-sales engineering
Role takeaway
- Fit
- AI scaffolding service invoked from a mastery-based CAL item engine after incorrect attempts.
- Architecture and integration
- Hook LLM support into attempt lifecycle events; store recovery metrics; disable bypass chat during gated practice.
- Prerequisites
- Item bank with mastery criteria
- Attempt logging API
- Content alignment to tested topics
- Constraints
- Evidence from one district research platform
- Marginal delayed effects on practiced material only
- Short one-week delay horizon
- Security
- Protect attempt transcripts and prompts as student education records; minimize training use of district data.
- Proof of value
- Next-attempt correctness after AI support vs CAL-only
- Attempts-to-recovery and time-on-support differentials
- Delayed practiced-item score difference under AI+mastery
Delivery
Role takeaway
- Work
- Configure mastery gates, enable post-error AI scaffolds, train teachers on interpreting slower-but-more-accurate progress, and run the delayed check.
- Dependencies
- Platform configuration rights
- Unit pacing calendar
- Assessment item quality
- Ownership
- Math curriculum lead with platform administrator; research partner for analysis.
- Skills and adoption
- Teachers must resist unlocking free chat that bypasses struggle; students need coaching to use AI for repair, not answers.
- Governance checkpoints
- Pre-pilot confirmation that unconstrained chat is disabled in practice mode
- Mid-unit review of recovery metrics
- Post-delayed-test go/no-go for expansion
- Acceptance criteria
- AI invoked only after incorrect attempts in pilot sections
- Documented mastery criteria active for the target topic
- Delayed practiced-item analysis completed and reviewed
- Risks
- Students use AI to avoid attempts
- Teachers disable mastery gates under pacing pressure
- Overgeneralizing marginal practiced-item gains to unpracticed transfer
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
AI help should be wired into post-error scaffolding and mastery progression engines (e.g., three-correct-in-a-row gates) rather than as a free-standing chatbot. Telemetry needs attempt-level correctness, time-on-support, and recovery metrics.
Governance
Who approves, reviews and stays accountable for outcomes?
Pilot protocols should pre-specify whether AI is allowed outside mastery constraints; unrestricted AI chat may undermine the mechanism the study identifies.
Security and privacy
What data, permissions and controls need testing?
Student problem-solving traces and AI support dialogues are education records; districts need purpose limitation when research platforms or vendors retain attempt logs.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Slower progression with higher accuracy may interact with IEP pacing accommodations; monitor whether mastery gates and AI scaffolds help or frustrate students who need alternate pathways.
Procurement
What should contracts, pricing and exit terms secure?
Prefer platforms that expose mastery and post-error AI hooks over generic chatbots; require vendors to support attempt-level evaluation exports for district research.
Operating model
Which teams own the service once it runs?
Instructional operating model must treat AI as part of the practice algorithm—owned by curriculum—not as a parallel consumer tool students open separately.
Publication history
- 2026-09-15K–12 · Issue 104 resources
Stable resource ID: numi-mastery-math-rct-2026