From the K–12 edition of November 10, 2026
NBER working paper: randomized trial of an AI math tutor with more than 6,000 Hamilton County, Tennessee middle-school students
National Bureau of Economic Research (Oreopoulos, Liut, Sungu and Low) · K-12 education (middle school mathematics) · Hamilton County, Tennessee, United States
- Publisher
- NBER
- Original publication
- August 2026
- Source retrieved
- 2026-11-11
- Event date
- 2026-03-23
What happened
Students were randomized to AI versus computer-assisted-learning-only support, mastery versus non-mastery progression and math topic. AI students attempted fewer questions and moved more slowly but were more accurate per attempt, and AI improved next-attempt correctness after mistakes. On a delayed test one week later, gains were modest: about three percentage points on the practiced Exercise 1 item for mastery students, only marginally significant, with no overall delayed-learning gain. Mastery alone did not improve delayed learning.
Why it matters
Newly relevant because the Nov. 10 Stanford SCALE review cites it as one of its U.S. K-12 studies; it gives districts a large, in-school, U.S. trial showing that a guard-railed AI tutor adds limited value and only within a structured workflow.
Evidence and measured results
Randomized field experiment in March-April 2026 across 20 schools and just under 100 teachers (grades 6-8), one 50-minute practice session and a delayed assessment one week later, on the NUMI computer-assisted learning platform; the full paper reports about 7,000 students. The headline delayed-test result is described by the authors as only marginally significant.
Limitations and uncertainty
Working paper, not peer reviewed. One short session and a one-week delay; the authors say it does not estimate effects of repeated use. The delayed-test effect is marginally significant and concentrated on one item. Platform-specific (NUMI), single-region study, and the tutor was a guard-railed design, so results may not carry over to general chatbots. Sample size is stated as more than 6,000 in the abstract.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-10; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Middle schools want scalable math tutoring but lack evidence that AI beats existing practice software on retained learning.
- Stakeholders
- Math curriculum leads
- Director of instruction
- Principals
- District technology and data office
- Discovery questions
- What practice platform do students already use and what delayed outcomes do you track?
- Does the candidate tutor guide reasoning after mistakes or reveal answers?
- How much practice time can classes allocate if AI slows pace?
- Value hypothesis
- Structured AI support after mistakes may add modest value over practice-only software; value depends on workflow design.
- Potential engagement
- Bounded pilot design with delayed-assessment measurement in one grade band and one math topic.
- Unsupported claims
- Transformational learning gains
- Gains from repeated use over a school year
- That results apply to open-ended chatbots
- Cost savings
Pre-sales engineering
Role takeaway
- Fit
- Fits districts evaluating AI hints or tutoring inside an existing math practice platform.
- Architecture and integration
- AI support embedded in progression and mastery rules with structured post-mistake prompts and item-level event logging.
- Prerequisites
- Practice platform with configurable progression
- Item-level telemetry
- Delayed assessment mechanism
- Teacher scripts
- Constraints
- One session and one-week follow-up only
- Platform-specific tutor design
- Marginal significance on key delayed result
- Security
- Study does not assess privacy; review student-data flows, retention and vendor access to logs before deployment.
- Proof of value
- Randomize AI versus practice-only at class level
- Measure next-attempt correctness after mistakes
- Administer a delayed unaided test a week later
Delivery
Role takeaway
- Work
- Run a controlled classroom pilot with a common script, comparison group and delayed assessment.
- Dependencies
- Teacher agreement to common script
- Class period for practice and for delayed test
- Platform logging access
- Ownership
- Math curriculum lead with district data office.
- Skills and adoption
- Teachers need to introduce the activity consistently and avoid answering content questions during the trial.
- Governance checkpoints
- Parent and privacy review before launch
- Review of pace and completion data after first sessions
- Go or no-go on delayed outcomes
- Acceptance criteria
- Proposed: delayed unaided accuracy at least equal to comparison
- Proposed: higher next-attempt correctness after mistakes in logs
- Proposed: completion rates acceptable given slower pace
- Risks
- Fewer questions completed with AI
- Over-reading a marginally significant result
- Short-session results not persisting
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
The tutor was embedded in a purpose-built practice platform with structured post-mistake prompts; value depended on workflow design, not a free-standing chatbot, so integration with progression rules and item-level logging matters.
Governance
Who approves, reviews and stays accountable for outcomes?
Districts should judge AI tutors on delayed unaided assessments rather than in-session performance, and require that tools turn mistakes into guided reasoning instead of answer reveal.
Security and privacy
What data, permissions and controls need testing?
The study does not assess privacy or data security. Platform telemetry on student work was central to analysis, so data collection, retention and vendor access terms need separate review.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Teachers followed a common script and did not answer math questions, so teacher role in real deployments is untested; student engagement with support is a key dependency.
Procurement
What should contracts, pricing and exit terms secure?
Request independent randomized evidence on delayed unaided outcomes and evidence of workflow-embedded design, and treat in-session accuracy as insufficient.
Operating model
Which teams own the service once it runs?
Assign classroom time for slower, guided practice and plan data ownership for platform telemetry.
Publication history
- 2026-11-10K–12 · Issue 653 resources
Stable resource ID: nber-w35621-hamilton-county-ai-math-tutoring-mastery-delayed-test-20260810