From the K–12 edition of September 15, 2026
Tennessee cluster RCT finds Khanmigo access gains resemble non-AI practice as engagement stays sparse
Annenberg Institute at Brown University / NBER (Oreopoulos & Low) · K-12 public education / education research · United States — Hamilton County Schools, Tennessee (18 middle schools)
- Publisher
- EdWorkingPapers / NBER
- Original publication
- August 2026 (EdWorkingPaper 26-1551; NBER Working Paper 35620)
- Source retrieved
- 2026-09-15
What happened
In a two-year cluster randomized trial across 18 Tennessee middle schools, assigning Khan Academy with Khanmigo (configured to coach rather than give answers) during daily remedial math raised achievement by about 1.3 national percentile ranks per term (roughly 0.06–0.08 SD over a school year), with an implied ~0.14 SD for a full year of active participation. Those gains resemble Khan Academy practice without AI assistance. Nearly all students tried the tutor once, but the median student messaged it on only a third of practice days and in only 17% of mistake sessions, usually with bare answers or suggested-prompt clicks—so engagement, not access alone, appears binding.
Why it matters
Directly relevant to U.S. district decisions on AI tutoring inside RTI/intervention blocks: procurement and pilot design should measure dialogue quality and sustained use, not license counts. Transferable caution for other SLED tutoring pilots claiming personalized AI coaches.
Evidence and measured results
Cluster RCT; AEA RCT Registry AEARCTR-0013519; University of Toronto REB. Intent-to-treat estimates from MAP/percentile outcomes; platform messaging logs document sparse substantive dialogue. Authors note gains resemble non-AI Khan Academy practice.
Limitations and uncertainty
Working paper; single Tennessee district; control condition may include other edtech; ITT averages over low AI use. Do not treat 0.14 SD active-participation implication as guaranteed for every rostered student.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-15; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Districts buying AI tutors expect personalized gains but may only get practice-software effects if students rarely converse with the tutor.
- Stakeholders
- Chief academic officer
- Math intervention / RTI lead
- Assessment & research director
- IT / instructional technology
- Discovery questions
- How will teachers prompt students to message the tutor after mistakes during intervention?
- What baseline growth do you already see from non-AI practice software?
- Which engagement metrics (not seat licenses) will count as success in a 90-day pilot?
- Value hypothesis
- If the district instruments and coaches for dialogue during existing RTI time, AI tutoring may deliver modest ITT gains on the order of strong practice tools—not automatic large tutoring-effect sizes.
- Potential engagement
- Bounded pilot in a subset of middle-school intervention sections with pre-registered MAP/percentile and dialogue-depth metrics versus business-as-usual remediation.
- Unsupported claims
- Guaranteed personal-tutor effect sizes for every licensed student
- AI access alone outperforming quality non-AI practice
- Districtwide savings from reduced human intervention staffing
Pre-sales engineering
Role takeaway
- Fit
- Coach-configured AI tutor embedded in daily remedial math practice platforms with rostering and teacher monitoring.
- Architecture and integration
- SIS roster sync, intervention-period scheduling, coach-not-answer model settings, and analytics distinguishing suggested-prompt clicks from student-authored math dialogue.
- Prerequisites
- Daily intervention time already on the master schedule
- Data-sharing / DPA with the tutoring vendor
- Teacher dashboard access and training
- Constraints
- Single-district Tennessee evidence base
- Effects sensitive to realized practice and messaging
- Working-paper status pending peer review
- Security
- Limit conversational logs to education purposes under the district DPA; review subprocessors and retention before enabling student prompts containing PII.
- Proof of value
- Compare ITT achievement change to non-AI practice baselines
- Track share of mistake sessions with a student message
- Audit that coach mode is enforced (no answer key dumps)
Delivery
Role takeaway
- Work
- Stand up rostered intervention sections, train teachers on post-error prompting, and operate a weekly engagement review with academic leaders.
- Dependencies
- Intervention schedule fidelity
- Platform rostering
- Assessment windows for growth measures
- Ownership
- Math intervention lead jointly with instructional technology; research office owns outcome analysis.
- Skills and adoption
- Teachers need routines to require tutor dialogue after errors; students need norms that bare answer submissions are insufficient.
- Governance checkpoints
- Pre-pilot DPA and ethics/research review
- Mid-pilot dialogue-rate gate before expansion
- End-of-term ITT growth review versus non-AI practice
- Acceptance criteria
- Documented coach-mode configuration across pilot schools
- Mistake-session messaging rate materially above the study’s 17% pattern or explicit remediation plan
- Pre-registered growth analysis delivered to academic leadership
- Risks
- Access without facilitation yields no AI-incremental benefit
- Teachers treat AI as unsupervised seatwork
- Overclaiming results to cut human tutoring capacity
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Tutor must be rostered into existing daily intervention time with coaching (not answer-giving) configuration, district dashboards, and logging that can distinguish clicks from mathematical dialogue. SSO/rostering and content alignment to remediation pathways matter more than novel model features if students never converse.
Governance
Who approves, reviews and stays accountable for outcomes?
District research partnerships and data-sharing agreements were prerequisites. Boards should require pre-registered evaluation metrics that include engagement depth, not only access or seat time, before scaling AI tutors districtwide.
Security and privacy
What data, permissions and controls need testing?
Study depended on a coordinated data-sharing agreement with the platform provider. Districts replicating should define education-record flows, retention, and subprocessors before enabling conversational tutors with student prompts.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Sample includes a substantial special-education share; districts should monitor whether sparse dialogue patterns differ for students with disabilities and whether coaching UI is accessible.
Procurement
What should contracts, pricing and exit terms secure?
Evaluate vendors on engagement analytics and coach-mode controls, not feature checklists alone; write success criteria into contracts around dialogue depth and learning outcomes.
Operating model
Which teams own the service once it runs?
AI tutoring must be operated as a facilitated intervention practice, with teacher time allocated to prompt and review dialogue—not as a set-and-forget license.
Publication history
- 2026-09-15K–12 · Issue 104 resources
Stable resource ID: khanmigo-tn-cluster-rct-2026