{"resourceId":"eef-nfer-maths-whizz-rct-2026","versions":[{"version":"external-a161c872bd38bf2b4dfd88259a278333208b055e693d3443101036004a4c4f7c","resource":{"id":"eef-nfer-maths-whizz-rct-2026","title":"EEF/NFER Maths-Whizz RCT: +1 month maths progress overall; +2 months for FSM pupils (g=0.08/0.14)","organization":"Education Endowment Foundation (evaluation by National Foundation for Educational Research)","sector":"K-12 primary mathematics intelligent tutoring evaluation","geography":"England, United Kingdom (63 primary schools; Years 2–5)","publishedAt":"June 2026 EEF evaluation report (NFER efficacy trial)","publicationDate":"2026-06-24","eventDate":"2025-07-01","sourceName":"Evaluation of Whizz Education: Maths-Whizz Intelligent Tutoring Programme (EEF PDF)","sourceLabel":"Independent EEF-commissioned interleaved RCT evaluation report","sourceUrl":"https://d2tic4wvo1iusb.cloudfront.net/production/documents/projects/maths_whizz_-_evaluation_report_v.1.0.0.pdf","evidenceClass":"independent-research","outcomeClass":"effective","topics":["developers-agents","accessibility-workforce","operating-model","knowledge-work"],"finding":"In an interleaved RCT across 63 English primary schools with 10,369 pupils (Years 2–5), allocation to Maths-Whizz—an AI-enabled adaptive tutoring programme intended for 45–60 minutes per week alongside usual maths teaching—produced an average of one additional month of maths progress on Renaissance Star Maths (effect size 0.08, 95% CI 0.02–0.14; n=8,983; p=0.01) with a high EEF security rating. FSM-eligible pupils gained about two additional months (effect size 0.14, 95% CI 0.06–0.22; n=2,301; p=0.01), though that subgroup estimate carries lower security due to smaller n. Average delivery cost was about £21.51 per pupil per year for two year groups. Authors highlight personalised scaffolding and gap-filling as plausible mechanisms for stronger FSM effects.","sledRelevance":"Provides scarce high-security causal evidence that a purpose-built adaptive AI maths tutor can raise primary attainment—with larger gains for disadvantaged pupils—informing U.S. district tutoring procurement and equity claims while requiring UK-to-U.S. transferability caveats.","evidence":"Opened the EEF CloudFront evaluation PDF end-to-end via curl and pdftotext. Verified Table 1 key conclusions, Table 2 effect sizes/security/cost ratings, sample (63 schools; 10,369 baseline pupils), dosage guidance (45–60 minutes/week), and high-security language for the primary finding.","architectureImplications":"Treat adaptive tutoring as a scheduled supplemental CAL workload (SSO, device density, ~0.5 Mbps per concurrent pupil) with dosage telemetry joinable to interim maths assessments—not as unmanaged general-purpose chat.","governanceImplications":"Pre-register Star-like interim outcomes and FSM/equity subgroup analyses; publish security/limitations slides matching independent evaluator standards before scaling licenses.","securityPrivacyImplications":"Pupil identifiers, usage logs, and assessment joins require DPA/minimisation; UK trial used UPNs and school-held data—U.S. districts must map to FERPA and state student-privacy rules before similar evaluation designs.","caveats":"UK primary efficacy trial (not U.S. K-12); vendor programme with school co-pay; FSM subgroup lower security; implementation fidelity and home access vary. Why now: first unarchived large independent AI-tutor RCT with equity-positive primary maths results, contemporaneous with U.S. homework-GenAI learning-cost and state AI-control debates.","streamIds":["k12"],"roles":{"sales":"Sales framing: districts want tutoring ROI with equity evidence, not chatbot demos. Lead with EEF’s high-security +1 month overall and +2 months FSM Star Maths results, disclose UK primary setting and roughly £21 per pupil per year cost, and ask whether the district can schedule 45–60 minutes weekly of adaptive practice with interim assessment joins. Position an independent evaluation readout—not vendor case studies—as the engagement artifact for board conversations.","engineering":"Engineering framing: instrument tutor versus practice modes, dosage minutes, and SSO-bound pupil identities joinable to interim maths scores. Architecture must support classroom concurrent sessions at modest bandwidth, teacher dashboards, and exportable midyear packets. Prove consumer GenAI stays out of band and that FSM or FRL subgroup analyses are reproducible from district-held data before claiming equity lift in any procurement dashboard or board slide.","delivery":"Delivery framing: stand up a year-group interleaved or matched pilot with professional development on engagement facilitation, capture weekly dosage telemetry, and run a pre-registered Star or MAP analysis with EEF-style limitations slides. Acceptance is a district-readable report showing months-of-progress equivalents, FSM or FRL contrasts, fidelity notes, and a formal go/no-go gate before any multi-year licensing expands beyond the pilot cohort."},"roleDetails":{"sales":{"customerProblem":"Districts lack independent causal evidence that adaptive AI maths tutoring improves attainment—especially for economically disadvantaged students.","stakeholders":["Superintendents","Chief academic officers","Title I / equity leads","CTOs","Board research liaisons"],"discoveryQuestions":["Can you schedule 45–60 minutes/week of adaptive maths practice without replacing core instruction?","Will you accept UK RCT transferability with a local interim replication?","How will you measure FSM/FRL subgroup effects on Star/MAP?","What independent evaluation standard will the board require before scale?"],"valueHypothesis":"Purpose-built adaptive AI tutoring can deliver roughly one month of maths progress overall and larger equity gains when dosage and interim assessments are measurable—without promising U.S. causal equivalence.","potentialEngagement":"One-year efficacy-style pilot: PD, dosage telemetry, and pre-registered interim maths analysis with published caveats.","unsupportedClaims":["Guaranteed U.S. causal gains identical to EEF results","That general-purpose chatbots produce the same equity effects","Cost savings that replace human tutoring entirely"]},"engineering":{"fit":"Fit for districts with device access, interim maths assessments, and capacity to schedule supplemental adaptive practice; poor fit where AI use is unmanaged consumer chat.","architectureIntegration":"Student IdP → adaptive tutor tenancy → dosage warehouse → roster join to interim maths assessments; teacher resource library optional.","prerequisites":["Devices and Wi-Fi for concurrent classroom sessions","Interim maths assessment extracts","Teacher PD time for facilitation sessions","Data-sharing agreement for dosage–score joins"],"constraints":["UK evidence base may not transfer to U.S. curricula","FSM subgroup estimate has lower security","Home access inequality can affect dosage"],"security":"Keep pupil identifiers and usage logs in enterprise tenancy; minimize PII in vendor exports; align with FERPA/state student-privacy statutes.","proofOfValue":["Reproduce dosage distributions vs 45–60 minute target","Replicate overall and FRL/FSM interim effect directions","Document high-security vs lower-security subgroup caveats","Show cost-per-pupil within district budget envelope"]},"delivery":{"work":"Onboard year groups, train teachers on engagement facilitation, monitor weekly dosage, administer endpoint interim maths, and brief leadership on months-of-progress and equity contrasts.","dependencies":["Assessment vendor accounts","School timetable slots","Vendor implementation support","Privacy review clearance"],"ownership":"Academic/teaching-and-learning lead owns outcomes; CTO owns platform and data joins; equity lead owns subgroup reporting.","skillsAdoption":"Teachers need short PD on facilitating engagement and interpreting dashboards; avoid treating the tutor as unsupervised homework replacement.","governanceCheckpoints":["Pre-registration of primary outcome and subgroups","Midyear dosage fidelity review","Endpoint limitations briefing before scale decision"],"acceptanceCriteria":["Target share of pupils hitting weekly dosage threshold","Published interim effect estimate with CI and caveats","FRL/FSM subgroup table delivered to board","Go/no-go memo signed by CAO and CTO"],"risks":["Attrition and test-day absence bias","Over-claiming UK results as U.S. proof","Dosage collapse when home access is weak"]}},"retrievedAt":"2026-09-21T03:14:20Z","enrichedAt":"2026-09-21T03:24:38Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Stronger FSM effects imply adaptive tutors may help close maths gaps if schools ensure device access and scheduled practice for disadvantaged pupils; PD must keep teachers facilitating engagement rather than offloading instruction.","procurementImplications":"Prefer independent efficacy evidence and per-pupil cost transparency; require dosage reporting, interim assessment joins, and equity subgroup analyses in contracts before multi-year commitments.","operatingModelImplications":"Operate as supplemental weekly tutoring blocks with fidelity monitoring—not as optional homework AI—so dosage, not license counts, drives the operating model.","sourceVerification":{"openedUrl":"https://d2tic4wvo1iusb.cloudfront.net/production/documents/projects/maths_whizz_-_evaluation_report_v.1.0.0.pdf","referenceExcerpt":"These findings have a high security rating. This was an efficacy trial, which tested whether","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}