{"resourceId":"google-eedi-expanded-uk-ai-tutoring-rct-1179-students-no-star-maths-gain-20261029","versions":[{"version":"external-83e71b9275e66fb9a566913b33af0e43c75c875c949aa04b13a5165b22d9ffc1","resource":{"id":"google-eedi-expanded-uk-ai-tutoring-rct-1179-students-no-star-maths-gain-20261029","title":"Google LearnLM team and Eedi: pre-registered RCT of 1,179 UK secondary students finds supervised AI tutoring matched expert human tutors in-lesson but neither beat standard remediation on Star Maths after 10 weeks","organization":"Google (LearnLM team) and Eedi","sector":"K-12 student-facing AI tutoring (secondary mathematics)","geography":"United Kingdom (10 state secondary schools)","publishedAt":"October 29, 2026 (technical report dated 2026-10-29; Eedi release same day)","publicationDate":"2026-10-29","eventDate":null,"sourceName":"Google LearnLM technical report","sourceLabel":"Vendor-authored pre-registered RCT technical report","sourceUrl":"https://services.google.com/fh/files/misc/learnlm_oct26.pdf","evidenceClass":"vendor-claim","outcomeClass":"mixed","topics":["knowledge-work","operating-model","accessibility-workforce"],"finding":"A pre-registered RCT (AEARCTR-0018079) across 10 UK state secondary schools and 58 classrooms assigned 1,179 students for a 10-week term to standard Eedi remediation, expert human tutoring, AI tutoring (Gemini with a pedagogical prompt), or AI tutoring with personalised learning-history context. Tutors reviewed every AI draft and approved 94.7% of 94,385 drafts unchanged; audits found maths errors in fewer than three per 1,000 AI messages, comparable to human tutors. On in-platform measures and a mid-trial quiz, both AI arms performed similarly to human tutors and better than standard remediation. On the externally scored Star Maths test, tutoring showed no overall advantage over standard remediation. Exploratory results favoured girls, while tutored boys trailed, and tutored students covered fewer study units.","sledRelevance":"Directly K-12 and new tonight: one of the larger school-based AI tutoring RCTs, published with subgroup and conversation-level detail. It separates assisted, in-the-moment performance from retained broad learning, which is the distinction the K-12 brief asks for.","evidence":"Pre-registered RCT, Bayesian analysis with 95% credible intervals; outcomes at three levels (in-platform, mid-trial quiz, Star Maths at baseline and endline; Star Maths analysis covers students who sat both tests). Each student kept one condition for the whole trial. Authored by the vendor teams that built the tutor.","architectureImplications":"The tutor is not an off-the-shelf chatbot: Gemini runs inside Eedi's constrained platform that supplies the diagnosed misconception and current question; adding personalised history did not clearly improve outcomes. Every message passed a human approval step.","governanceImplications":"Interpretation: districts judging AI tutors should ask for broad, externally scored outcomes and subgroup results, not only in-session accuracy. The authors themselves call for full-year trials before guiding policy.","securityPrivacyImplications":"Limited relevance: the report discusses safety review of messages (no harmful content reached students) but this run did not inspect data-handling terms; conversation data from minors was analysed by the vendors.","caveats":"Vendor-authored, not peer reviewed. Ten weeks only; one all-boys school meant boys were 66.1% of the sample; gender findings are exploratory. Human-in-the-loop supervision of every message is costly and differs from the human-on-the-loop model Eedi is now testing. UK context limits transfer to U.S. districts.","streamIds":["k12"],"roles":{"sales":"The framing point is that the best-run AI tutor trials now show in-lesson gains that have not yet reached broad test scores. This Google and Eedi RCT of 1,179 UK students found AI tutoring matched expert humans in the moment but did not beat standard remediation on Star Maths after 10 weeks. A bounded offer is helping a district design a pilot with a broad outcome measure. The evidence cannot support claims that AI tutoring raises standardized scores or replaces human tutors.","engineering":"Fit is a constrained AI tutor embedded in a diagnostic maths platform, not a general chatbot. Interpretation: the useful design lessons are supplying the misconception and question context to the model, routing every draft through a reviewer during early use, and logging conversations for audits of errors and tone. Extra learning-history context did not clearly help, so engineers should not assume more student data improves results. Supervision workload rose when one tutor monitored several students at once.","delivery":"Delivery work is running a measured tutoring pilot that tracks both near-term and broad outcomes. A district maths or assessment office would own the evaluation, with teachers present during sessions as in this trial. Interpretation: acceptance criteria could include a pre-registered plan, a broad standardized measure at baseline and endline, subgroup reporting by gender and prior attainment, and a run of at least a full term. Risks include slower curriculum coverage, disengaged students, and reviewer fatigue."},"roleDetails":{"sales":{"customerProblem":"Districts want scalable tutoring, but evidence that AI tutoring improves broad learning, not just in-session answers, is still thin.","stakeholders":["District maths and curriculum leaders","Assessment and research office","Teachers supervising sessions","Families","Tutoring vendors"],"discoveryQuestions":["What broad outcome measure will a pilot use at baseline and endline?","Will results be reported by gender and prior attainment?","Who reviews AI messages, and how many students per reviewer?","How long can the pilot run?"],"valueHypothesis":"A constrained AI tutor may match human tutors on in-session remediation at lower marginal cost; broad learning gains are unproven.","potentialEngagement":"Evaluation design support for a one-term AI tutoring pilot with a standardized outcome and subgroup reporting.","unsupportedClaims":["That AI tutoring raises standardized test scores","That AI tutoring can replace human tutors","That results transfer unchanged to U.S. schools"]},"engineering":{"fit":"Constrained, misconception-aware AI tutor inside a maths practice platform.","architectureIntegration":"Interpretation: pass the diagnosed misconception and current item to the model, keep a reviewer approval path, and log all conversations for audit.","prerequisites":["Diagnostic item bank with misconception tags","Reviewer staffing during early use","Conversation logging and audit process"],"constraints":["Student privacy law in the deploying jurisdiction","Reviewer workload when supervising several students","Vendor-controlled model updates"],"security":"Confirm retention and secondary use of student conversation data, and that safety audits cover tone and maths errors.","proofOfValue":["Audit a sample of AI messages for maths errors and tone","Compare in-session gains with a broad standardized measure","Check results by gender and prior attainment"]},"delivery":{"work":"Run a teacher-present AI tutoring pilot with pre-registered measures at near-term and broad levels.","dependencies":["Baseline and endline assessment","Reviewer or tutor staffing","Teacher scheduling of weekly sessions"],"ownership":"District maths or assessment office with participating school leaders.","skillsAdoption":"Reviewers need training on fast approval decisions; teachers need guidance for students who disengage or ask to stop.","governanceCheckpoints":["Pre-registration of measures","Mid-term safety and engagement review","Board briefing before scale-up"],"acceptanceCriteria":["Broad outcome measured at baseline and endline","Subgroup results reported","Error rate in audited AI messages tracked against human tutors"],"risks":["In-session gains without broad learning","Uneven effects by gender","Reduced curriculum coverage"]}},"retrievedAt":"2026-10-29T21:59:33-05:00","enrichedAt":"2026-10-29T22:04:48-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Tutors found AI drafts helpful and time-saving but supervising them less rewarding than tutoring directly; typed chat may favour confident readers and writers.","procurementImplications":"Interpretation: contracts for AI tutoring should require outcome and safety reporting similar to what this trial published; cost was not reported.","operatingModelImplications":"Human-in-the-loop approval of every message was used here; the vendor is moving to human-on-the-loop oversight in a 15,000-student year-long trial.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://services.google.com/fh/files/misc/learnlm_oct26.pdf","referenceExcerpt":"Unexpectedly, we did not detect such clear benefits on Star Maths","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}