{"resourceId":"hechinger-khanmigo-engagement-gap-20260921","versions":[{"version":"external-e8fed93b6ebdb719171224e4df6308af006964c6d7f0bc9aab41fed1b565857c","resource":{"id":"hechinger-khanmigo-engagement-gap-20260921","title":"Hechinger: Tennessee Khanmigo study finds AI tutor added no gains over non-AI practice as students refused Socratic help","organization":"The Hechinger Report","sector":"K-12 education journalism / independent research coverage","geography":"United States (Tennessee middle-school remediation trial)","publishedAt":"September 21, 2026","publicationDate":"2026-09-21","eventDate":null,"sourceName":"The Hechinger Report","sourceLabel":"Hechinger Report Proof Points","sourceUrl":"https://hechingerreport.org/proof-points-khanmigo-math-ai-tutor/","evidenceClass":"independent-reporting","outcomeClass":"cautionary","topics":["knowledge-work","accessibility-workforce","operating-model"],"finding":"Jill Barshay reports that a University of Toronto two-year study in 18 Tennessee middle schools found students assigned Khan Academy with Khanmigo improved modestly versus usual remediation, but gains matched prior non-AI Khan Academy practice because students largely stopped using the Socratic tutor after it withheld answers and asked questions. Sal Khan publicly accepted the engagement diagnosis and described product changes (in-workflow help after errors; optional AI-off).","sledRelevance":"Directly informs US district procurement of student-facing AI tutors: access and 'Socratic' framing are insufficient without sustained engagement and unaided transfer evidence.","evidence":"Independent education journalism summarizing NBER/EdWorkingPapers RCT results and on-record Khan Academy response; not a substitute for the primary paper already archived elsewhere.","architectureImplications":"Prefer in-path help-after-struggle over optional AI tabs; support AI-off modes; capture dialogue-depth telemetry.","governanceImplications":"Pilot contracts should define engagement thresholds and unaided outcome metrics before scaling student AI tutors.","securityPrivacyImplications":"Even 'safe' tutoring bots need DPAs, content filters, and admin disablement; engagement failure is not a privacy success metric.","caveats":"Journalistic secondary source; remediation-block population; product versions evolving; primary RCT already in SLED archive under edworkingpapers URL.","streamIds":["k12"],"roles":{"sales":"Sales framing: do not sell 'AI tutor access' as an achievement intervention for behind-grade students. Hechinger’s 21 Sep coverage of the Tennessee Khanmigo cluster RCT shows Khan Academy practice helped modestly, but the Socratic AI assistant added no extra gains because students largely refused hints after it withheld answers. Lead with engagement design, teacher orchestration, and unaided transfer metrics—not license counts. Interpretation only; limited to reported middle-school remediation context.","engineering":"Engineering framing: treat optional AI-tutor tabs as a high-abandonment path. The Hechinger-reported Tennessee trial found nearly all students tried Khanmigo initially—often seeking answers—then disengaged when the bot asked questions instead. Prefer in-workflow triggers after incorrect attempts, configurable AI-off modes, and telemetry on dialogue depth—not just clicks or session starts. Architecture must still satisfy local privacy, age filters, and supervision rules; interpretation only for remediation settings described in the coverage.","delivery":"Delivery framing: operationalize 'help after struggle' with teacher routines, not a soft-launch email. Hechinger notes Khan Academy is changing prompts and considering credit for bot-assisted redos—districts must decide whether that credit still measures independent mastery. Schedule a mid-pilot go/no-go on engagement thresholds and unaided quizzes before any scale decision. Interpretation for SLED implementation planning in middle-school remediation blocks."},"roleDetails":{"sales":{"customerProblem":"Districts buying student AI tutors for remediation risk paying for access that students abandon when tools refuse to give answers.","stakeholders":["assistant superintendent curriculum","MTSS/intervention leads","math department chairs","IT/edtech procurement","board academic committee"],"discoveryQuestions":["What share of assigned students initiate and sustain multi-turn AI tutor dialogues after wrong answers?","Do your outcome contracts require unaided post-test gains, not platform engagement minutes?","How will teachers orchestrate AI help inside the remediation block versus leaving it optional?","Can the product be configured so help appears after struggle without becoming an answer machine?"],"valueHypothesis":"If districts procure AI tutoring only with engagement-gated pilots and unaided transfer measures, they can avoid overpaying for unused Socratic features that add no incremental learning.","potentialEngagement":"4–8 week remediation pilot comparing AI-on vs AI-off within the same practice platform, with session-level dialogue analytics and pre/post unaided items.","unsupportedClaims":["Guaranteed grade-level catch-up from enabling an AI tutor","That any Socratic chatbot will outperform non-AI practice by design","Universal student willingness to engage with questioning tutors"]},"engineering":{"fit":"Fits districts already using a practice platform that can A/B AI assistant on/off inside the same content graph.","architectureIntegration":"Embed help after incorrect attempts in the learning path; avoid separate optional tabs; export dialogue-turn and hint-acceptance metrics to the data warehouse.","prerequisites":["Rostered SSO into the practice platform","Ability to disable AI assistant per school or class","Teacher visibility into student AI dialogue volume","Baseline assessment for remediation placement"],"constraints":["Low voluntary engagement with questioning tutors limits effect sizes","Remediation-block findings may not transfer to core class periods","Product versions change; 2023-era features differ from later in-workflow designs"],"security":"Keep student dialogue under district DPA; no training on student chats; age-appropriate filters; teacher/admin kill capability.","proofOfValue":["Compare AI-on vs AI-off cohorts on unaided unit quizzes","Track median substantive dialogue turns per student per week","Blind review of a sample of AI hints for accuracy"]},"delivery":{"work":"Stand up AI-on/AI-off remediation pilot, teacher facilitation protocol, weekly engagement dashboard, and end-of-pilot unaided assessment comparison.","dependencies":["platform AI feature flags","intervention schedule fidelity","assessment warehouse access","family notice where required"],"ownership":"Curriculum/intervention lead owns learning outcomes; IT owns SSO and logging; principals own block fidelity.","skillsAdoption":"Coach teachers to require attempt-before-help and to treat AI dialogue as formative evidence, not grading credit by default.","governanceCheckpoints":["Pre-pilot privacy/DPA review","Week-3 engagement threshold review","End-of-pilot board or cabinet briefing on AI incremental value"],"acceptanceCriteria":["≥X% of students complete ≥N substantive AI turns/week or AI feature is paused","Unaided post-test delta for AI-on not worse than AI-off","Zero critical safety incidents in sampled dialogues"],"risks":["Students treat AI as answer machine workarounds","Teachers award mastery credit for AI-assisted redos without transfer checks","Vendor product changes invalidate pilot assumptions"]}},"retrievedAt":"2026-09-27T22:08:15-05:00","enrichedAt":"2026-09-27T22:09:30-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Behind-grade learners may least engage with questioning tutors; teacher orchestration and accessibility supports must be designed, not assumed.","procurementImplications":"Buy practice platforms with measurable AI incremental value and kill-switches; avoid paying premium solely for unused AI assistants.","operatingModelImplications":"Remediation blocks need facilitation protocols; optional AI help is an operating-model choice, not a set-and-forget feature.","sourceVerification":{"openedUrl":"https://hechingerreport.org/proof-points-khanmigo-math-ai-tutor/","referenceExcerpt":"Khan Academy appeared to help. But the Khanmigo tutor didn’t add anything.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}