{"resourceId":"khanmigo-tn-cluster-rct-2026","versions":[{"version":"external-f7c07cd1072741493061eb522eec5890314394fcdd6303d64c4b32d2cfa9d0c9","resource":{"id":"khanmigo-tn-cluster-rct-2026","title":"Tennessee cluster RCT finds Khanmigo access gains resemble non-AI practice as engagement stays sparse","organization":"Annenberg Institute at Brown University / NBER (Oreopoulos & Low)","sector":"K-12 public education / education research","geography":"United States — Hamilton County Schools, Tennessee (18 middle schools)","publishedAt":"August 2026 (EdWorkingPaper 26-1551; NBER Working Paper 35620)","publicationDate":"2026-08-01","eventDate":null,"sourceName":"EdWorkingPapers / NBER","sourceLabel":"Oreopoulos & Low Khanmigo two-year school experiment","sourceUrl":"https://edworkingpapers.com/sites/default/files/ai26-1551.pdf","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","accessibility-workforce","operating-model"],"finding":"In a two-year cluster randomized trial across 18 Tennessee middle schools, assigning Khan Academy with Khanmigo (configured to coach rather than give answers) during daily remedial math raised achievement by about 1.3 national percentile ranks per term (roughly 0.06–0.08 SD over a school year), with an implied ~0.14 SD for a full year of active participation. Those gains resemble Khan Academy practice without AI assistance. Nearly all students tried the tutor once, but the median student messaged it on only a third of practice days and in only 17% of mistake sessions, usually with bare answers or suggested-prompt clicks—so engagement, not access alone, appears binding.","sledRelevance":"Directly relevant to U.S. district decisions on AI tutoring inside RTI/intervention blocks: procurement and pilot design should measure dialogue quality and sustained use, not license counts. Transferable caution for other SLED tutoring pilots claiming personalized AI coaches.","evidence":"Cluster RCT; AEA RCT Registry AEARCTR-0013519; University of Toronto REB. Intent-to-treat estimates from MAP/percentile outcomes; platform messaging logs document sparse substantive dialogue. Authors note gains resemble non-AI Khan Academy practice.","architectureImplications":"Tutor must be rostered into existing daily intervention time with coaching (not answer-giving) configuration, district dashboards, and logging that can distinguish clicks from mathematical dialogue. SSO/rostering and content alignment to remediation pathways matter more than novel model features if students never converse.","governanceImplications":"District research partnerships and data-sharing agreements were prerequisites. Boards should require pre-registered evaluation metrics that include engagement depth, not only access or seat time, before scaling AI tutors districtwide.","securityPrivacyImplications":"Study depended on a coordinated data-sharing agreement with the platform provider. Districts replicating should define education-record flows, retention, and subprocessors before enabling conversational tutors with student prompts.","caveats":"Working paper; single Tennessee district; control condition may include other edtech; ITT averages over low AI use. Do not treat 0.14 SD active-participation implication as guaranteed for every rostered student.","streamIds":["k12"],"roles":{"sales":"Frame discovery around whether the district’s math intervention block can produce sustained student–tutor dialogue, not whether an AI tutor license is available. Use this RCT to set bounded expectations: modest ITT gains similar to strong practice software are plausible, while transformative personal-tutor claims are unsupported when messaging is sparse. Engage curriculum, intervention, and research/assessment owners together, and propose a pilot that instruments mistake-session help-seeking rather than vanity adoption metrics.","engineering":"Fit is strongest as coach-mode tutoring inside an existing CAL/intervention platform with rostering and teacher dashboards. Integrate with SIS/LMS schedules for daily RTI periods, capture message-level telemetry (not only logins), and disable answer-giving modes that collapse dialogue into answer dumps. Proof of value should compare treated vs control growth and dialogue rates against the paper’s sparse-use pattern before architecture expansion.","delivery":"Delivery work centers on teacher routines that prompt students to message after errors, professional learning on coaching norms, and weekly review of engagement funnels. Operational ownership sits with intervention leads plus instructional technology. Acceptance should include minimum dialogue coverage on mistake sessions and MAP/percentile growth checks, with risks that access without facilitation reproduces null AI-incremental effects."},"roleDetails":{"sales":{"customerProblem":"Districts buying AI tutors expect personalized gains but may only get practice-software effects if students rarely converse with the tutor.","stakeholders":["Chief academic officer","Math intervention / RTI lead","Assessment & research director","IT / instructional technology"],"discoveryQuestions":["How will teachers prompt students to message the tutor after mistakes during intervention?","What baseline growth do you already see from non-AI practice software?","Which engagement metrics (not seat licenses) will count as success in a 90-day pilot?"],"valueHypothesis":"If the district instruments and coaches for dialogue during existing RTI time, AI tutoring may deliver modest ITT gains on the order of strong practice tools—not automatic large tutoring-effect sizes.","potentialEngagement":"Bounded pilot in a subset of middle-school intervention sections with pre-registered MAP/percentile and dialogue-depth metrics versus business-as-usual remediation.","unsupportedClaims":["Guaranteed personal-tutor effect sizes for every licensed student","AI access alone outperforming quality non-AI practice","Districtwide savings from reduced human intervention staffing"]},"engineering":{"fit":"Coach-configured AI tutor embedded in daily remedial math practice platforms with rostering and teacher monitoring.","architectureIntegration":"SIS roster sync, intervention-period scheduling, coach-not-answer model settings, and analytics distinguishing suggested-prompt clicks from student-authored math dialogue.","prerequisites":["Daily intervention time already on the master schedule","Data-sharing / DPA with the tutoring vendor","Teacher dashboard access and training"],"constraints":["Single-district Tennessee evidence base","Effects sensitive to realized practice and messaging","Working-paper status pending peer review"],"security":"Limit conversational logs to education purposes under the district DPA; review subprocessors and retention before enabling student prompts containing PII.","proofOfValue":["Compare ITT achievement change to non-AI practice baselines","Track share of mistake sessions with a student message","Audit that coach mode is enforced (no answer key dumps)"]},"delivery":{"work":"Stand up rostered intervention sections, train teachers on post-error prompting, and operate a weekly engagement review with academic leaders.","dependencies":["Intervention schedule fidelity","Platform rostering","Assessment windows for growth measures"],"ownership":"Math intervention lead jointly with instructional technology; research office owns outcome analysis.","skillsAdoption":"Teachers need routines to require tutor dialogue after errors; students need norms that bare answer submissions are insufficient.","governanceCheckpoints":["Pre-pilot DPA and ethics/research review","Mid-pilot dialogue-rate gate before expansion","End-of-term ITT growth review versus non-AI practice"],"acceptanceCriteria":["Documented coach-mode configuration across pilot schools","Mistake-session messaging rate materially above the study’s 17% pattern or explicit remediation plan","Pre-registered growth analysis delivered to academic leadership"],"risks":["Access without facilitation yields no AI-incremental benefit","Teachers treat AI as unsupervised seatwork","Overclaiming results to cut human tutoring capacity"]}},"retrievedAt":"2026-09-15T18:49:10.771Z","enrichedAt":"2026-09-15T18:49:10.771Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Sample includes a substantial special-education share; districts should monitor whether sparse dialogue patterns differ for students with disabilities and whether coaching UI is accessible.","procurementImplications":"Evaluate vendors on engagement analytics and coach-mode controls, not feature checklists alone; write success criteria into contracts around dialogue depth and learning outcomes.","operatingModelImplications":"AI tutoring must be operated as a facilitated intervention practice, with teacher time allocated to prompt and review dialogue—not as a set-and-forget license.","sourceVerification":{"openedUrl":"https://edworkingpapers.com/sites/default/files/ai26-1551.pdf","referenceExcerpt":"term, or about 0.06 to 0.08 standard deviations over a school year; the implied effect of a full year of active","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}