From the K–12 edition of September 27, 2026
Hechinger: Tennessee Khanmigo study finds AI tutor added no gains over non-AI practice as students refused Socratic help
The Hechinger Report · K-12 education journalism / independent research coverage · United States (Tennessee middle-school remediation trial)
- Publisher
- The Hechinger Report
- Original publication
- September 21, 2026
- Source retrieved
- 2026-09-28
What happened
Jill Barshay reports that a University of Toronto two-year study in 18 Tennessee middle schools found students assigned Khan Academy with Khanmigo improved modestly versus usual remediation, but gains matched prior non-AI Khan Academy practice because students largely stopped using the Socratic tutor after it withheld answers and asked questions. Sal Khan publicly accepted the engagement diagnosis and described product changes (in-workflow help after errors; optional AI-off).
Why it matters
Directly informs US district procurement of student-facing AI tutors: access and 'Socratic' framing are insufficient without sustained engagement and unaided transfer evidence.
Evidence and measured results
Independent education journalism summarizing NBER/EdWorkingPapers RCT results and on-record Khan Academy response; not a substitute for the primary paper already archived elsewhere.
Limitations and uncertainty
Journalistic secondary source; remediation-block population; product versions evolving; primary RCT already in SLED archive under edworkingpapers URL.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-27; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Districts buying student AI tutors for remediation risk paying for access that students abandon when tools refuse to give answers.
- Stakeholders
- assistant superintendent curriculum
- MTSS/intervention leads
- math department chairs
- IT/edtech procurement
- board academic committee
- Discovery questions
- What share of assigned students initiate and sustain multi-turn AI tutor dialogues after wrong answers?
- Do your outcome contracts require unaided post-test gains, not platform engagement minutes?
- How will teachers orchestrate AI help inside the remediation block versus leaving it optional?
- Can the product be configured so help appears after struggle without becoming an answer machine?
- Value hypothesis
- If districts procure AI tutoring only with engagement-gated pilots and unaided transfer measures, they can avoid overpaying for unused Socratic features that add no incremental learning.
- Potential engagement
- 4–8 week remediation pilot comparing AI-on vs AI-off within the same practice platform, with session-level dialogue analytics and pre/post unaided items.
- Unsupported claims
- Guaranteed grade-level catch-up from enabling an AI tutor
- That any Socratic chatbot will outperform non-AI practice by design
- Universal student willingness to engage with questioning tutors
Pre-sales engineering
Role takeaway
- Fit
- Fits districts already using a practice platform that can A/B AI assistant on/off inside the same content graph.
- Architecture and integration
- Embed help after incorrect attempts in the learning path; avoid separate optional tabs; export dialogue-turn and hint-acceptance metrics to the data warehouse.
- Prerequisites
- Rostered SSO into the practice platform
- Ability to disable AI assistant per school or class
- Teacher visibility into student AI dialogue volume
- Baseline assessment for remediation placement
- Constraints
- Low voluntary engagement with questioning tutors limits effect sizes
- Remediation-block findings may not transfer to core class periods
- Product versions change; 2023-era features differ from later in-workflow designs
- Security
- Keep student dialogue under district DPA; no training on student chats; age-appropriate filters; teacher/admin kill capability.
- Proof of value
- Compare AI-on vs AI-off cohorts on unaided unit quizzes
- Track median substantive dialogue turns per student per week
- Blind review of a sample of AI hints for accuracy
Delivery
Role takeaway
- Work
- Stand up AI-on/AI-off remediation pilot, teacher facilitation protocol, weekly engagement dashboard, and end-of-pilot unaided assessment comparison.
- Dependencies
- platform AI feature flags
- intervention schedule fidelity
- assessment warehouse access
- family notice where required
- Ownership
- Curriculum/intervention lead owns learning outcomes; IT owns SSO and logging; principals own block fidelity.
- Skills and adoption
- Coach teachers to require attempt-before-help and to treat AI dialogue as formative evidence, not grading credit by default.
- Governance checkpoints
- Pre-pilot privacy/DPA review
- Week-3 engagement threshold review
- End-of-pilot board or cabinet briefing on AI incremental value
- Acceptance criteria
- ≥X% of students complete ≥N substantive AI turns/week or AI feature is paused
- Unaided post-test delta for AI-on not worse than AI-off
- Zero critical safety incidents in sampled dialogues
- Risks
- Students treat AI as answer machine workarounds
- Teachers award mastery credit for AI-assisted redos without transfer checks
- Vendor product changes invalidate pilot assumptions
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Prefer in-path help-after-struggle over optional AI tabs; support AI-off modes; capture dialogue-depth telemetry.
Governance
Who approves, reviews and stays accountable for outcomes?
Pilot contracts should define engagement thresholds and unaided outcome metrics before scaling student AI tutors.
Security and privacy
What data, permissions and controls need testing?
Even 'safe' tutoring bots need DPAs, content filters, and admin disablement; engagement failure is not a privacy success metric.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Behind-grade learners may least engage with questioning tutors; teacher orchestration and accessibility supports must be designed, not assumed.
Procurement
What should contracts, pricing and exit terms secure?
Buy practice platforms with measurable AI incremental value and kill-switches; avoid paying premium solely for unused AI assistants.
Operating model
Which teams own the service once it runs?
Remediation blocks need facilitation protocols; optional AI help is an operating-model choice, not a set-and-forget feature.
Publication history
- 2026-09-27K–12 · Issue 224 resources
Stable resource ID: hechinger-khanmigo-engagement-gap-20260921