From the K–12 edition of October 19, 2026
Large rural China RCT: an AI-driven adaptive math program did no better than non-adaptive software or no software for 8,647 primary students
Yue Ma, Tianli Feng, Robert Fairlie, Chengfang Liu, Prashant Loyalka, Scott Rozelle, Xinwu Zhang (Stanford, UESTC, UCLA/NBER, Peking University, NUIST); CESifo Working Paper 12837 · K-12 (primary grades 4–6) AI-personalized computer-assisted learning in mathematics · Sichuan Province, China (105 rural primary schools in eight low-income counties); international evidence with limited U.S. transferability
- Publisher
- RePEc/IDEAS listing of CESifo Working Paper 12837
- Original publication
- CESifo Working Paper 12837 (original July 2026; this version September 2026, PDF dated September 25, 2026); journal version in Economics of Education Review (DOI 10.1016/j.econedurev.2026.102850) indexed as published October 17, 2026; journal page returned HTTP 403 to this agent
- Source retrieved
- 2026-10-20
What happened
Researchers randomly assigned 315 grade 4–6 classes (8,647 students) in 105 rural Sichuan schools to an AI-driven adaptive math program (AI-CAL), a non-adaptive program matched on content, time, hardware, gamification and feedback (R-CAL), or no CAL. Students used CAL 80 minutes a week for 11 weeks (March–June 2021). In the preferred specification AI-CAL vs R-CAL was −0.040 SD (not significant; effects above 0.01 SD ruled out) and AI-CAL vs pure control was −0.036 SD (SE 0.027). Attrition was under 1% per arm. Teachers cited older hardware and low student motivation. The authors conclude paying a premium for adaptive features is hard to justify in these settings. Why now: the peer-reviewed journal version was indexed as published October 17 and surfaced after last night's research, and it is one of the few large causal tests of AI personalization.
Why it matters
Directly relevant to district purchasing of 'AI-personalized' practice software: the strongest new causal result this week is a precisely estimated null.
Evidence and measured results
Pre-registered cluster RCT with active and pure controls, cluster-robust standard errors, minimal attrition and a replication package. Outcome is a curriculum-aligned math test at the end of 11 weeks; no long-term retention measured.
Limitations and uncertainty
Intervention ran in 2021 and tested pre-generative adaptive learning, not chatbot or LLM tutors. Rural, under-resourced Chinese primary schools; results may not transfer to U.S. districts or higher-engagement settings. Heterogeneity analyses were exploratory. The two programs were different products, not one program with adaptivity switched off. Journal version not opened (ScienceDirect returned HTTP 403); findings come from the CESifo working paper.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-19; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Districts pay for AI-personalized learning software without evidence that personalization beats simpler, cheaper practice tools in their setting.
- Stakeholders
- Chief academic officer
- Math curriculum director
- Assessment and research office
- Procurement
- School board
- Discovery questions
- Which products do you pay a premium for because they are adaptive or AI-driven?
- What comparison did vendors use to show impact?
- Do you have a non-adaptive alternative you could test against?
- How old is the device fleet students use for practice?
- Value hypothesis
- A structured evidence review and comparison pilot could avoid paying for personalization that adds no learning value; savings are not demonstrated by this source.
- Potential engagement
- An eight-week evidence and pilot-design review for one AI-personalized math product, ending in a comparison plan and renewal recommendation.
- Unsupported claims
- Does not show generative AI tutors are ineffective.
- Does not transfer directly to U.S. districts.
- No cost data reported.
Pre-sales engineering
Role takeaway
- Fit
- Fits districts evaluating or renewing adaptive or AI-personalized practice platforms.
- Architecture and integration
- Configure pilots so the same content runs with and without adaptive sequencing, and capture item-level logs the study lacked.
- Prerequisites
- Device and lab capacity inventory
- Rostering for comparison groups
- Agreed outcome assessment
- Constraints
- 2021 adaptive software, not generative AI
- Rural China setting
- 11-week outcome window
- Security
- Limit diagnostic and behavioral data collection to what the evaluation needs and confirm vendor retention terms.
- Proof of value
- Randomized or matched comparison against non-adaptive option
- Item-level logs confirm students saw comparable content
- Retention measured after the pilot ends
Delivery
Role takeaway
- Work
- Embed a comparison evaluation in any AI-personalized math rollout before scaling.
- Dependencies
- Research or assessment office capacity
- Vendor data access
- Teacher and principal agreement to comparison groups
- Ownership
- District assessment and research office with the math curriculum director.
- Skills and adoption
- Train school leaders to read effect sizes and distinguish engagement from learning.
- Governance checkpoints
- Pre-registered outcome and comparison approved
- Mid-pilot implementation check
- Renewal decision tied to results
- Acceptance criteria
- Proposed: comparison report delivered before renewal
- Proposed: outcome measured at end and after a delay
- Proposed: decision documented against the evidence
- Risks
- Pilots too short to detect retention
- Engagement metrics mistaken for learning
- Hardware limits confound results
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
The adaptive engine was a commercial production system running on school lab computers; older hardware was a reported constraint. Interpretation: adaptivity alone is not a reliable source of value, so architecture choices should not presume a premium for personalization engines.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports requiring independent, setting-matched evidence before paying for AI personalization, and comparing against simpler non-adaptive alternatives rather than against nothing.
Security and privacy
What data, permissions and controls need testing?
Not addressed in the study. Interpretation: adaptive systems collect continuous diagnostic data; if they add no learning value, that data collection deserves extra scrutiny.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Teachers reported low student motivation with gamified software; benefits did not differ by baseline achievement, gender or family wealth in exploratory analyses.
Procurement
What should contracts, pricing and exit terms secure?
Authors argue paying a premium for adaptive features over simpler CAL is hard to justify in these settings; vendors should be asked for setting-matched independent evidence.
Operating model
Which teams own the service once it runs?
Personalization software is not a substitute for teacher instruction or infrastructure readiness; evaluation should precede scale.
Publication history
- 2026-10-19K–12 · Issue 435 resources
Stable resource ID: china-ai-cal-rct-null-effect-rural-primary-math-2026