{"resourceId":"learnlm-uk-eedi-exploratory-rct-202512","versions":[{"version":"external-f490f7f3c4d53e3be19fc8eed0afbf7a8ed9e7e7535ed746f6edc0d94345c7a0","resource":{"id":"learnlm-uk-eedi-exploratory-rct-202512","title":"LearnLM exploratory RCT (UK): human-supervised pedagogically fine-tuned tutoring matches human tutors and lifts novel-problem success +5.5 pp","organization":"LearnLM Team (Google) & Eedi","sector":"K-12 secondary mathematics tutoring research (vendor-affiliated exploratory RCT)","geography":"United Kingdom (five secondary schools)","publishedAt":"arXiv preprint 29 December 2025 (arXiv:2512.23633v1)","publicationDate":"2025-12-29","eventDate":null,"sourceName":"AI tutoring can safely and effectively support students: An exploratory RCT in UK classrooms (PDF)","sourceLabel":"Exploratory randomized controlled trial preprint (Google/Eedi)","sourceUrl":"https://arxiv.org/pdf/2512.23633","evidenceClass":"academic-research","outcomeClass":"effective","topics":["knowledge-work","developers-agents","operating-model","accessibility-workforce"],"finding":"In an exploratory RCT with 165 students across five UK secondary schools, expert tutors supervised LearnLM drafts on the Eedi math platform and could edit before sending. Tutors approved 76.4% of drafts with zero or minimal edits. Students supported by LearnLM performed at least as well as human-tutor-only peers on measured outcomes and were 5.5 percentage points more likely to solve novel problems on subsequent topics (66.2% vs 60.7%). Authors frame results as evidence for pedagogically fine-tuned, human-gated tutoring—not unsupervised chatbots—and call for longer longitudinal and standardized-assessment follow-up.","sledRelevance":"Transferable design constraint for U.S. districts running narrow HS AI tutoring pilots: require coach-mode models, human edit gates, and unaided novel-problem checks rather than access-only deployments.","evidence":"Opened arXiv PDF via curl+pdftotext: abstract sample and design, supervision protocol, 76.4% approval statistic, +5.5 pp novel-problem contrast, tutor interview themes on Socratic questioning, and limitations calling for longitudinal standardized outcomes.","architectureImplications":"Tutoring products should expose draft-then-human-approve workflows, pedagogy-tuned system prompts/models, and session logs; student-facing autonomous answer-giving modes are out of scope for this evidence.","governanceImplications":"Pilot approvals should mandate human supervision ratios, edit-rate monitoring, and pre-registered unaided transfer measures; vendor-authored studies need independent replication plans.","securityPrivacyImplications":"Chat transcripts and tutor edits are sensitive student data—require DPAs, retention limits, and no training on district student content unless contractually controlled.","caveats":"Exploratory, N=165, UK secondary math only; Google & Eedi authorship; human tutors always present—does not justify unsupervised student GenAI; Dec 2025 preprint used now because it maps directly onto 2026–27 supervised-pilot operating models (e.g., NYCPS).","streamIds":["k12"],"roles":{"sales":"Sales framing: sell supervised coach-mode tutoring with human edit gates and novel-problem evaluation, citing +5.5 pp transfer under supervision—not ‘AI replaces tutors.’ Discover whether the district can staff tutor oversight; if not, this evidence does not apply.","engineering":"Engineering framing: build draft→approve message pipelines, measure zero-edit rates, block answer-dumping, and instrument unaided delayed items. Do not ship unattended student chat based on this paper.","delivery":"Delivery framing: pilot with trained tutor supervisors, weekly edit-rate reviews, and a pre-registered novel-problem endpoint. Success is supervised efficacy with safety audits—not seat licenses alone."},"roleDetails":{"sales":{"customerProblem":"Districts want scalable tutoring evidence without repeating unsupervised GenAI learning harms.","stakeholders":["Chief academic officer","math curriculum leads","intervention/RTI directors","CTOs","research/evaluation"],"discoveryQuestions":["Can every AI tutor message be human-approved before students see it?","What unaided novel-problem or MAP-like endpoint will you pre-register?","What tutor staffing ratio is sustainable for supervision?","Is the model constrained to coach rather than answer?"],"valueHypothesis":"Bounded supervised pilots with pedagogy-tuned models can match human tutoring on short-run transfer when edit gates are real.","potentialEngagement":"8–12 week supervised tutoring pilot with edit-rate dashboards and a novel-problem outcome card.","unsupportedClaims":["That unsupervised chatbots produce the same gains","Guaranteed MAP/state-test gains from this exploratory RCT","U.S. elementary generalizability"]},"engineering":{"fit":"Fit for platforms that can insert human tutors in the loop; poor fit for fully autonomous homework bots.","architectureIntegration":"LLM draft service → tutor review UI → student chat; analytics on edit distance; content filters for safety.","prerequisites":["Tutor workforce","math item bank with delayed novel items","privacy-reviewed logging"],"constraints":["UK secondary context","vendor authorship","exploratory sample size"],"security":"Encrypt transcripts; restrict vendor training use; role-based tutor access.","proofOfValue":["≥70% drafts approved with minimal edits in pilot sample","Unaided novel-problem lift vs human-only control","Zero critical safety incidents in audited message sample"]},"delivery":{"work":"Recruit/train supervising tutors, configure coach-mode, run RCT-like pilot arms, audit messages, and report transfer outcomes.","dependencies":["School scheduling","tutor contracts","IRB/research review if needed","platform admins"],"ownership":"Academic intervention lead with research partner; platform vendor supports but does not grade success alone.","skillsAdoption":"Tutor training on when to rewrite AI drafts and how to keep Socratic scaffolds.","governanceCheckpoints":["Protocol lock","midpoint safety audit","endpoint analysis","go/no-go for scale"],"acceptanceCriteria":["Human approval on 100% of student-visible messages","Pre-registered novel-problem analysis completed","Public limitations statement including authorship"],"risks":["Supervision decays into rubber-stamping","Overclaiming beyond exploratory design","Student data reused for model training contrary to DPA"]}},"retrievedAt":"2026-09-22T03:15:55Z","enrichedAt":"2026-09-22T03:20:19Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Requires trained human tutors; accessibility depends on platform a11y and whether supervision workload is funded.","procurementImplications":"Contracts should require human-in-the-loop controls, no-training clauses on student chats, and independent evaluation rights.","operatingModelImplications":"Tutoring AI is an assisted-workflow staffing model, not a set-and-forget license.","sourceVerification":{"openedUrl":"https://arxiv.org/pdf/2512.23633","referenceExcerpt":"students who received support from LearnLM were 5.5 percentage points more likely to solve novel problems on subsequent topics","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}