{"resourceId":"china-ai-cal-rct-null-effect-rural-primary-math-2026","versions":[{"version":"external-1c4ce0fd849594df0ab773df6f492538c7ede168e8601c214680ff8144b7f0ed","resource":{"id":"china-ai-cal-rct-null-effect-rural-primary-math-2026","title":"Large rural China RCT: an AI-driven adaptive math program did no better than non-adaptive software or no software for 8,647 primary students","organization":"Yue Ma, Tianli Feng, Robert Fairlie, Chengfang Liu, Prashant Loyalka, Scott Rozelle, Xinwu Zhang (Stanford, UESTC, UCLA/NBER, Peking University, NUIST); CESifo Working Paper 12837","sector":"K-12 (primary grades 4–6) AI-personalized computer-assisted learning in mathematics","geography":"Sichuan Province, China (105 rural primary schools in eight low-income counties); international evidence with limited U.S. transferability","publishedAt":"CESifo Working Paper 12837 (original July 2026; this version September 2026, PDF dated September 25, 2026); journal version in Economics of Education Review (DOI 10.1016/j.econedurev.2026.102850) indexed as published October 17, 2026; journal page returned HTTP 403 to this agent","publicationDate":"2026-09-25","eventDate":null,"sourceName":"RePEc/IDEAS listing of CESifo Working Paper 12837","sourceLabel":"Pre-registered randomized controlled trial (AEARCTR-0007888); working paper with full methods","sourceUrl":"https://ideas.repec.org/p/ces/ceswps/_12837.html","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["governance-procurement","infrastructure","operating-model"],"finding":"Researchers randomly assigned 315 grade 4–6 classes (8,647 students) in 105 rural Sichuan schools to an AI-driven adaptive math program (AI-CAL), a non-adaptive program matched on content, time, hardware, gamification and feedback (R-CAL), or no CAL. Students used CAL 80 minutes a week for 11 weeks (March–June 2021). In the preferred specification AI-CAL vs R-CAL was −0.040 SD (not significant; effects above 0.01 SD ruled out) and AI-CAL vs pure control was −0.036 SD (SE 0.027). Attrition was under 1% per arm. Teachers cited older hardware and low student motivation. The authors conclude paying a premium for adaptive features is hard to justify in these settings. Why now: the peer-reviewed journal version was indexed as published October 17 and surfaced after last night's research, and it is one of the few large causal tests of AI personalization.","sledRelevance":"Directly relevant to district purchasing of 'AI-personalized' practice software: the strongest new causal result this week is a precisely estimated null.","evidence":"Pre-registered cluster RCT with active and pure controls, cluster-robust standard errors, minimal attrition and a replication package. Outcome is a curriculum-aligned math test at the end of 11 weeks; no long-term retention measured.","architectureImplications":"The adaptive engine was a commercial production system running on school lab computers; older hardware was a reported constraint. Interpretation: adaptivity alone is not a reliable source of value, so architecture choices should not presume a premium for personalization engines.","governanceImplications":"Supports requiring independent, setting-matched evidence before paying for AI personalization, and comparing against simpler non-adaptive alternatives rather than against nothing.","securityPrivacyImplications":"Not addressed in the study. Interpretation: adaptive systems collect continuous diagnostic data; if they add no learning value, that data collection deserves extra scrutiny.","caveats":"Intervention ran in 2021 and tested pre-generative adaptive learning, not chatbot or LLM tutors. Rural, under-resourced Chinese primary schools; results may not transfer to U.S. districts or higher-engagement settings. Heterogeneity analyses were exploratory. The two programs were different products, not one program with adaptivity switched off. Journal version not opened (ScienceDirect returned HTTP 403); findings come from the CESifo working paper.","streamIds":["k12"],"roles":{"sales":"Sales framing: districts are being sold 'AI-personalized' practice as a learning upgrade, and this pre-registered trial of 8,647 students found no benefit over simpler software or none. That supports a bounded evidence-review offer that helps buyers compare adaptive products with cheaper non-adaptive options using setting-matched studies. Do not claim AI tutoring fails in general: the study tested 2021 adaptive software in rural China, not generative AI tutors in U.S. classrooms.","engineering":"Engineering framing: the adaptive engine itself added no measurable value while hardware age and student motivation constrained use. Pre-sales should test a district's device fleet and lab capacity before assuming software features matter, and should design pilots that compare adaptive and non-adaptive configurations of the same content. Proof of value means a randomized or matched comparison against a simpler option, not usage dashboards.","delivery":"Delivery framing: build evaluation into any AI-personalized math rollout. Define a comparison group, a pre-registered outcome and a decision point before scale. The district's assessment or research office owns the design with curriculum leaders. Acceptance means a completed comparison report that informs renewal. Risks include short pilots that miss retention effects and treating engagement metrics as learning."},"roleDetails":{"sales":{"customerProblem":"Districts pay for AI-personalized learning software without evidence that personalization beats simpler, cheaper practice tools in their setting.","stakeholders":["Chief academic officer","Math curriculum director","Assessment and research office","Procurement","School board"],"discoveryQuestions":["Which products do you pay a premium for because they are adaptive or AI-driven?","What comparison did vendors use to show impact?","Do you have a non-adaptive alternative you could test against?","How old is the device fleet students use for practice?"],"valueHypothesis":"A structured evidence review and comparison pilot could avoid paying for personalization that adds no learning value; savings are not demonstrated by this source.","potentialEngagement":"An eight-week evidence and pilot-design review for one AI-personalized math product, ending in a comparison plan and renewal recommendation.","unsupportedClaims":["Does not show generative AI tutors are ineffective.","Does not transfer directly to U.S. districts.","No cost data reported."]},"engineering":{"fit":"Fits districts evaluating or renewing adaptive or AI-personalized practice platforms.","architectureIntegration":"Configure pilots so the same content runs with and without adaptive sequencing, and capture item-level logs the study lacked.","prerequisites":["Device and lab capacity inventory","Rostering for comparison groups","Agreed outcome assessment"],"constraints":["2021 adaptive software, not generative AI","Rural China setting","11-week outcome window"],"security":"Limit diagnostic and behavioral data collection to what the evaluation needs and confirm vendor retention terms.","proofOfValue":["Randomized or matched comparison against non-adaptive option","Item-level logs confirm students saw comparable content","Retention measured after the pilot ends"]},"delivery":{"work":"Embed a comparison evaluation in any AI-personalized math rollout before scaling.","dependencies":["Research or assessment office capacity","Vendor data access","Teacher and principal agreement to comparison groups"],"ownership":"District assessment and research office with the math curriculum director.","skillsAdoption":"Train school leaders to read effect sizes and distinguish engagement from learning.","governanceCheckpoints":["Pre-registered outcome and comparison approved","Mid-pilot implementation check","Renewal decision tied to results"],"acceptanceCriteria":["Proposed: comparison report delivered before renewal","Proposed: outcome measured at end and after a delay","Proposed: decision documented against the evidence"],"risks":["Pilots too short to detect retention","Engagement metrics mistaken for learning","Hardware limits confound results"]}},"retrievedAt":"2026-10-19T22:10:31-05:00","enrichedAt":"2026-10-19T22:18:02-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Teachers reported low student motivation with gamified software; benefits did not differ by baseline achievement, gender or family wealth in exploratory analyses.","procurementImplications":"Authors argue paying a premium for adaptive features over simpler CAL is hard to justify in these settings; vendors should be asked for setting-matched independent evidence.","operatingModelImplications":"Personalization software is not a substitute for teacher instruction or infrastructure readiness; evaluation should precede scale.","sourceVerification":{"openedUrl":"https://ideas.repec.org/p/ces/ceswps/_12837.html","referenceExcerpt":"Results indicate that AI-CAL does not significantly improve achievement, with estimates precise enough to rule out non-trivial positive effects.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}