{"resourceId":"nber-w35621-hamilton-county-ai-math-tutoring-mastery-delayed-test-20260810","versions":[{"version":"external-62cf11f5355e81fd53bd6073fe3926046d49bbd2f5594b722ffedc330a4aba5d","resource":{"id":"nber-w35621-hamilton-county-ai-math-tutoring-mastery-delayed-test-20260810","title":"NBER working paper: randomized trial of an AI math tutor with more than 6,000 Hamilton County, Tennessee middle-school students","organization":"National Bureau of Economic Research (Oreopoulos, Liut, Sungu and Low)","sector":"K-12 education (middle school mathematics)","geography":"Hamilton County, Tennessee, United States","publishedAt":"August 2026","publicationDate":null,"eventDate":"2026-03-23","sourceName":"NBER","sourceLabel":"NBER working paper","sourceUrl":"https://www.nber.org/papers/w35621","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","operating-model"],"finding":"Students were randomized to AI versus computer-assisted-learning-only support, mastery versus non-mastery progression and math topic. AI students attempted fewer questions and moved more slowly but were more accurate per attempt, and AI improved next-attempt correctness after mistakes. On a delayed test one week later, gains were modest: about three percentage points on the practiced Exercise 1 item for mastery students, only marginally significant, with no overall delayed-learning gain. Mastery alone did not improve delayed learning.","sledRelevance":"Newly relevant because the Nov. 10 Stanford SCALE review cites it as one of its U.S. K-12 studies; it gives districts a large, in-school, U.S. trial showing that a guard-railed AI tutor adds limited value and only within a structured workflow.","evidence":"Randomized field experiment in March-April 2026 across 20 schools and just under 100 teachers (grades 6-8), one 50-minute practice session and a delayed assessment one week later, on the NUMI computer-assisted learning platform; the full paper reports about 7,000 students. The headline delayed-test result is described by the authors as only marginally significant.","architectureImplications":"The tutor was embedded in a purpose-built practice platform with structured post-mistake prompts; value depended on workflow design, not a free-standing chatbot, so integration with progression rules and item-level logging matters.","governanceImplications":"Districts should judge AI tutors on delayed unaided assessments rather than in-session performance, and require that tools turn mistakes into guided reasoning instead of answer reveal.","securityPrivacyImplications":"The study does not assess privacy or data security. Platform telemetry on student work was central to analysis, so data collection, retention and vendor access terms need separate review.","caveats":"Working paper, not peer reviewed. One short session and a one-week delay; the authors say it does not estimate effects of repeated use. The delayed-test effect is marginally significant and concentrated on one item. Platform-specific (NUMI), single-region study, and the tutor was a guard-railed design, so results may not carry over to general chatbots. Sample size is stated as more than 6,000 in the abstract.","streamIds":["k12"],"roles":{"sales":"Use this as a grounded counterweight to tutoring hype: a large U.S. trial found AI added value over practice-only software mainly by improving recovery from mistakes, with modest and marginally significant delayed results. Frame offers around structured practice workflows, teacher routines and measurement, not transformational gains. Ask prospects whether their intended tool shows hints and reasoning after mistakes or just gives answers. Do not cite the three-point result as a general effect; it was one item after one session.","engineering":"Check whether a candidate tutor is embedded in progression logic and gives structured post-mistake support rather than open answers. The study shows slower pace and fewer questions with AI, so throughput metrics can mislead. Require item-level telemetry, delayed-assessment capability and ability to configure mastery rules. Test hint design, since the key mechanism was guided walkthroughs after errors. Student-data handling and vendor access to telemetry need their own review because the trial does not address them.","delivery":"Design a pilot with a comparison group, a delayed unaided check about a week after practice, and teacher scripts that keep students working silently first. Expect slower pace; schedule time accordingly. Define success on delayed accuracy rather than questions completed. Treat results as modest and uncertain, and plan multi-week follow-up because the paper tested one session. Assign a data owner for platform logs and agree retention before launch."},"roleDetails":{"sales":{"customerProblem":"Middle schools want scalable math tutoring but lack evidence that AI beats existing practice software on retained learning.","stakeholders":["Math curriculum leads","Director of instruction","Principals","District technology and data office"],"discoveryQuestions":["What practice platform do students already use and what delayed outcomes do you track?","Does the candidate tutor guide reasoning after mistakes or reveal answers?","How much practice time can classes allocate if AI slows pace?"],"valueHypothesis":"Structured AI support after mistakes may add modest value over practice-only software; value depends on workflow design.","potentialEngagement":"Bounded pilot design with delayed-assessment measurement in one grade band and one math topic.","unsupportedClaims":["Transformational learning gains","Gains from repeated use over a school year","That results apply to open-ended chatbots","Cost savings"]},"engineering":{"fit":"Fits districts evaluating AI hints or tutoring inside an existing math practice platform.","architectureIntegration":"AI support embedded in progression and mastery rules with structured post-mistake prompts and item-level event logging.","prerequisites":["Practice platform with configurable progression","Item-level telemetry","Delayed assessment mechanism","Teacher scripts"],"constraints":["One session and one-week follow-up only","Platform-specific tutor design","Marginal significance on key delayed result"],"security":"Study does not assess privacy; review student-data flows, retention and vendor access to logs before deployment.","proofOfValue":["Randomize AI versus practice-only at class level","Measure next-attempt correctness after mistakes","Administer a delayed unaided test a week later"]},"delivery":{"work":"Run a controlled classroom pilot with a common script, comparison group and delayed assessment.","dependencies":["Teacher agreement to common script","Class period for practice and for delayed test","Platform logging access"],"ownership":"Math curriculum lead with district data office.","skillsAdoption":"Teachers need to introduce the activity consistently and avoid answering content questions during the trial.","governanceCheckpoints":["Parent and privacy review before launch","Review of pace and completion data after first sessions","Go or no-go on delayed outcomes"],"acceptanceCriteria":["Proposed: delayed unaided accuracy at least equal to comparison","Proposed: higher next-attempt correctness after mistakes in logs","Proposed: completion rates acceptable given slower pace"],"risks":["Fewer questions completed with AI","Over-reading a marginally significant result","Short-session results not persisting"]}},"retrievedAt":"2026-11-10T21:59:46-06:00","enrichedAt":"2026-11-10T21:59:46-06:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Teachers followed a common script and did not answer math questions, so teacher role in real deployments is untested; student engagement with support is a key dependency.","procurementImplications":"Request independent randomized evidence on delayed unaided outcomes and evidence of workflow-embedded design, and treat in-session accuracy as insufficient.","operatingModelImplications":"Assign classroom time for slower, guided practice and plan data ownership for platform telemetry.","peerReview":"unknown","sourceVerification":{"openedUrl":"https://www.nber.org/papers/w35621","referenceExcerpt":"AI students progressed more slowly and attempted fewer questions, but answered more accurately conditional on reaching an attempt.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}