{"resourceId":"numi-mastery-math-rct-2026","versions":[{"version":"external-f7c07cd1072741493061eb522eec5890314394fcdd6303d64c4b32d2cfa9d0c9","resource":{"id":"numi-mastery-math-rct-2026","title":"NUMI field experiment ties LLM tutoring value to mastery workflows that recover from mistakes","organization":"Annenberg Institute at Brown University (Oreopoulos, Liut, Sungu & Low)","sector":"K-12 public education / education research","geography":"United States — Hamilton County Schools, Tennessee middle schools","publishedAt":"August 17, 2026 (EdWorkingPaper 26-1552)","publicationDate":"2026-08-17","eventDate":null,"sourceName":"EdWorkingPapers","sourceLabel":"Making AI Tutoring Productive mastery experiment","sourceUrl":"https://edworkingpapers.com/sites/default/files/ai26-1552.pdf","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","operating-model","developers-agents"],"finding":"A randomized field experiment with more than 6,000 middle-school students on the NUMI CAL platform crossed AI versus CAL-only support with mastery versus non-mastery progression. AI students progressed more slowly and attempted fewer questions but were more accurate when they attempted; the clearest mechanism was better next-attempt correctness and fewer attempts to recover after mistakes, with more time on structured support. Mastery raised three-correct-in-a-row attainment but did not alone improve delayed learning. The most encouraging delayed-test signal appeared when AI was embedded in the mastery workflow, with marginally significant gains concentrated on practiced material—supporting the claim that LLM tutoring adds value over standard CAL only with structure that turns mistakes into productive learning.","sledRelevance":"Helps districts interpret mixed AI tutoring pilots: product selection and classroom workflow must enforce productive struggle and mastery gates, not open-ended chat. Relevant to instructional design RFPs and professional learning for AI-supported practice.","evidence":"Registered field experiment (AEA RCT Registry AEARCTR-0018678) with delayed assessment about one week later covering practiced and unpracticed items; factorial randomization across AI, mastery, and topics.","architectureImplications":"AI help should be wired into post-error scaffolding and mastery progression engines (e.g., three-correct-in-a-row gates) rather than as a free-standing chatbot. Telemetry needs attempt-level correctness, time-on-support, and recovery metrics.","governanceImplications":"Pilot protocols should pre-specify whether AI is allowed outside mastery constraints; unrestricted AI chat may undermine the mechanism the study identifies.","securityPrivacyImplications":"Student problem-solving traces and AI support dialogues are education records; districts need purpose limitation when research platforms or vendors retain attempt logs.","caveats":"Delayed outcomes short; encouraging AI+mastery results are marginally significant and concentrated on practiced Exercise 1 content. Research platform (NUMI) may differ from commercial suites. Same district partnership as the Khanmigo RCT.","streamIds":["k12"],"roles":{"sales":"Position AI tutoring as a mastery-workflow enhancer, not a standalone chatbot seat. Use this experiment to challenge buyers who want unrestricted student GenAI during practice: the measured mechanism is structured recovery after mistakes. Discovery should map whether the district’s practice platform can gate progression and invoke AI only in productive-struggle states, and the engagement should be a factorial or A/B pilot comparing AI+mastery versus AI without mastery gates.","engineering":"Fit requires attempt-level integration: detect errors, invoke scaffolded LLM support, measure next-attempt success, and enforce mastery criteria before topic advancement. Avoid architectures that let students bypass items via unconstrained chat. Proof of value should replicate delayed practiced-item checks and recovery-time metrics, acknowledging marginal significance in the source.","delivery":"Implement classroom norms and platform settings that keep AI inside mastery pathways; train teachers to interpret slower progression with higher accuracy as an expected pattern. Ownership spans curriculum platform admins and math leads. Acceptance criteria should include recovery-after-mistake KPIs and a delayed quiz on practiced content, with risk that removing mastery gates erases AI incremental value."},"roleDetails":{"sales":{"customerProblem":"Districts see uneven AI tutoring results because tools are bolted on without mastery structure that forces productive recovery from errors.","stakeholders":["Curriculum & instruction","Math department chairs","Edtech platform owner","Research/assessment"],"discoveryQuestions":["Can your practice platform enforce mastery gates and post-error scaffolds today?","Do students currently use AI to skip struggle rather than repair misconceptions?","What delayed-practice assessment will you run one week after a unit?"],"valueHypothesis":"LLM support can beat standard CAL when embedded in mastery workflows that increase productive time after mistakes—without guaranteeing broad transfer gains.","potentialEngagement":"Short factorial classroom pilot comparing AI+mastery versus AI without mastery on one math topic with a delayed practiced-item check.","unsupportedClaims":["Unrestricted AI chat improves delayed learning by itself","Mastery badges alone guarantee learning gains","Guaranteed acceleration of pacing for all students"]},"engineering":{"fit":"AI scaffolding service invoked from a mastery-based CAL item engine after incorrect attempts.","architectureIntegration":"Hook LLM support into attempt lifecycle events; store recovery metrics; disable bypass chat during gated practice.","prerequisites":["Item bank with mastery criteria","Attempt logging API","Content alignment to tested topics"],"constraints":["Evidence from one district research platform","Marginal delayed effects on practiced material only","Short one-week delay horizon"],"security":"Protect attempt transcripts and prompts as student education records; minimize training use of district data.","proofOfValue":["Next-attempt correctness after AI support vs CAL-only","Attempts-to-recovery and time-on-support differentials","Delayed practiced-item score difference under AI+mastery"]},"delivery":{"work":"Configure mastery gates, enable post-error AI scaffolds, train teachers on interpreting slower-but-more-accurate progress, and run the delayed check.","dependencies":["Platform configuration rights","Unit pacing calendar","Assessment item quality"],"ownership":"Math curriculum lead with platform administrator; research partner for analysis.","skillsAdoption":"Teachers must resist unlocking free chat that bypasses struggle; students need coaching to use AI for repair, not answers.","governanceCheckpoints":["Pre-pilot confirmation that unconstrained chat is disabled in practice mode","Mid-unit review of recovery metrics","Post-delayed-test go/no-go for expansion"],"acceptanceCriteria":["AI invoked only after incorrect attempts in pilot sections","Documented mastery criteria active for the target topic","Delayed practiced-item analysis completed and reviewed"],"risks":["Students use AI to avoid attempts","Teachers disable mastery gates under pacing pressure","Overgeneralizing marginal practiced-item gains to unpracticed transfer"]}},"retrievedAt":"2026-09-15T18:49:10.771Z","enrichedAt":"2026-09-15T18:49:10.771Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Slower progression with higher accuracy may interact with IEP pacing accommodations; monitor whether mastery gates and AI scaffolds help or frustrate students who need alternate pathways.","procurementImplications":"Prefer platforms that expose mastery and post-error AI hooks over generic chatbots; require vendors to support attempt-level evaluation exports for district research.","operatingModelImplications":"Instructional operating model must treat AI as part of the practice algorithm—owned by curriculum—not as a parallel consumer tool students open separately.","sourceVerification":{"openedUrl":"https://edworkingpapers.com/sites/default/files/ai26-1552.pdf","referenceExcerpt":"tutoring can add value over standard CAL, but its value depends on structure that turns mistakes into","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}