{"resourceId":"frontiers-genai-learning-meta-analysis-wide-prediction-interval-small-study-bias-20261021","versions":[{"version":"external-79c770cc812db4a9c29dd084ca0cb712191496bf6a32ea325905545ae0fd89b3","resource":{"id":"frontiers-genai-learning-meta-analysis-wide-prediction-interval-small-study-bias-20261021","title":"Frontiers meta-analysis of 58 GenAI learning studies finds large pooled effect but a prediction interval that includes zero and likely small-study inflation; only 7 primary-school effects","organization":"Frontiers in Education (peer-reviewed review article)","sector":"K-12 and higher education: generative AI effects on learning, cognitive and linguistic outcomes","geography":"International (studies from multiple countries)","publishedAt":"21 October 2026 (Frontiers in Education, Digital Education section)","publicationDate":"2026-10-21","eventDate":null,"sourceName":"Frontiers in Education","sourceLabel":"Peer-reviewed meta-analysis","sourceUrl":"https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2026.1971476/full","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","governance-procurement"],"finding":"A three-level random-effects meta-analysis of 68 effect sizes from 58 experimental or quasi-experimental studies (6,941 participants, published 2023-2026) found a pooled g = 0.864 (95% CI 0.720-1.009) for GenAI on learning-related outcomes, with I² about 88% and a 95% prediction interval of -0.213 to 1.942, meaning a new setting could see null or negative effects. Egger's test showed funnel asymmetry and PET-PEESE suggested the true effect may be g ≈ 0.26-0.45. Most effects came from higher education (k = 50); secondary k = 11 (g = 0.612) and primary k = 7 (g = 1.311). Educational-level and outcome-domain moderators did not survive Holm correction.","sledRelevance":"K-12 relevant as a caution: headline effect sizes are often quoted to districts, but K-12 evidence is thin (18 of 68 effects) and the authors say local results could be null. Newly relevant because districts such as Bloomington are designing their own outcome studies this week.","evidence":"Peer-reviewed meta-analysis; English-language peer-reviewed studies only; many primary studies are small and language-focused; cultural-orientation contrasts are confounded with level and subject.","architectureImplications":"Limited relevance; tool type (ChatGPT-based vs other) did not reliably moderate effects.","governanceImplications":"Interpretation: districts should treat vendor-cited pooled effects as averages that do not predict local outcomes and require local measurement.","securityPrivacyImplications":"Not addressed by the study.","caveats":"Dominated by higher-education samples; K-12 subgroups are small; publication-bias corrections diverge (trim-and-fill vs PET-PEESE); between-study variance hit a boundary estimate. Published Oct. 21, before the last run; included because unarchived and newly relevant to district pilot design.","streamIds":["k12"],"roles":{"sales":"The framing point is that the research average is not a promise. This meta-analysis finds a large pooled effect of generative AI on learning, but its prediction interval includes zero and bias corrections cut the estimate to roughly 0.26-0.45. For K-12 buyers, a bounded offer is helping districts read effect-size claims and set local success measures. The evidence cannot support guaranteed learning gains in any district, especially at K-12, where only 18 effect sizes exist.","engineering":"Fit is evaluation design rather than architecture. The study found that tool type did not reliably change effects, so platform choice matters less than how use is structured and measured. Interpretation: pre-sales engineers should propose pilots with comparison groups, pre-specified outcomes and usage logging so a district can tell whether it landed in the positive or null part of the prediction interval. The thin primary-school evidence means claims for young children need extra care.","delivery":"Delivery work is building measurement into AI rollouts. District assessment or research staff would own outcome tracking with vendors supplying usage data. Interpretation: acceptance criteria could include a baseline, comparison classrooms, effect estimates with intervals and a decision rule for scaling or stopping. Risks include over-reading small pilots, publication-style bias in internal reporting where only good results are shared, and confusing engagement metrics with learning."},"roleDetails":{"sales":{"customerProblem":"District leaders hear large AI effect sizes but face uncertain local results and thin K-12 evidence.","stakeholders":["Chief academic officers","District research and assessment staff","School boards","Curriculum leaders"],"discoveryQuestions":["What effect-size claims have vendors cited, and from what grade levels?","How will the district measure learning outcomes locally?","What result would cause the district to stop a pilot?"],"valueHypothesis":"Setting local success measures upfront could prevent scaling tools that do not work locally; this is a hypothesis, not a measured result.","potentialEngagement":"A short evaluation-design engagement for an AI pilot, including comparison groups and decision rules.","unsupportedClaims":["That GenAI produces a 0.86 SD gain in any given school","That effects are larger for primary students (k = 7, not significant after correction)","That a specific tool is superior"]},"engineering":{"fit":"Measurement and logging design for AI pilots.","architectureIntegration":"Interpretation: capture usage logs linked to pseudonymous student identifiers so outcomes can be compared across conditions.","prerequisites":["Baseline assessment data","Comparison group design","Data-sharing terms"],"constraints":["Small samples limit power","Privacy rules on linking usage to achievement","Short pilot windows"],"security":"Use pseudonymized data and minimum necessary fields for outcome analysis.","proofOfValue":["Report local effect with confidence interval","Compare against the 0.26-0.45 bias-corrected range","Check results separately for subgroups"]},"delivery":{"work":"Embed outcome measurement and decision rules into AI pilot plans.","dependencies":["Assessment calendar","Research staff capacity","Vendor usage data"],"ownership":"District research and assessment office with curriculum leadership.","skillsAdoption":"Leaders need basic literacy in effect sizes, confidence and prediction intervals.","governanceCheckpoints":["Pre-registered pilot plan","Mid-pilot fidelity check","Board review of results before scaling"],"acceptanceCriteria":["Baseline and comparison data collected","Effect estimate reported with interval","Scale or stop decision documented"],"risks":["Over-reading small pilots","Selective reporting","Engagement mistaken for learning"]}},"retrievedAt":"2026-10-27T22:05:55-05:00","enrichedAt":"2026-10-27T22:10:39-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Not addressed directly; authors note younger learners depend more on external scaffolding.","procurementImplications":"Interpretation: procurement should ask vendors for evidence by grade level rather than pooled effects.","operatingModelImplications":"Supports routine local evaluation as part of AI adoption.","sourceVerification":{"openedUrl":"https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2026.1971476/full","referenceExcerpt":"the wide prediction interval and evidence of small-study inflation counsel caution in generalizing the point estimate","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}