{"resourceId":"medly-gcse-micro-rct-20260913","versions":[{"version":"external-f7c07cd1072741493061eb522eec5890314394fcdd6303d64c4b32d2cfa9d0c9","resource":{"id":"medly-gcse-micro-rct-20260913","title":"UK GCSE science micro-RCT reports preliminary AI tutoring gains while highlighting evaluation lag and attrition","organization":"What Worked Education / Durham University (Harrison et al.)","sector":"Secondary education research","geography":"United Kingdom — English secondary schools (GCSE Biology, Chemistry, Physics)","publishedAt":"2026-09-13 (arXiv 2609.14789v1)","publicationDate":"2026-09-13","eventDate":null,"sourceName":"arXiv","sourceLabel":"Evaluating AI Tutoring at the Speed of Innovation (Medly micro-RCT)","sourceUrl":"https://arxiv.org/pdf/2609.14789","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","governance-procurement","operating-model"],"finding":"A four-week multisite individually randomised evaluation of Medly, an AI tutoring platform, in GCSE science found primary ITT Hedges’ g = 0.33 (95% CI 0.18–0.48) versus business-as-usual self-directed revision among students with post-tests (644 of 929 baseline). Positive subject estimates and no clear Pupil Premium differential are reported, but authors label results preliminary because attrition was 30.7%, outcomes were curriculum-aligned rather than standardised, follow-up was short, and process evaluation response was limited. They argue educational AI needs rapid, repeatable micro-RCTs because interventions drift while conventional trials lag.","sledRelevance":"Transferable scrutiny for U.S. districts: even positive AI tutoring estimates can be fragile under attrition and short horizons, and product version drift can obsolete slow evaluations. Supports building cumulative local randomised tests into edtech adoption—not only one-off pilots.","evidence":"Individually randomised multisite trial; ITT with Hedges’ g; baseline n=929; post-test n=644; subject-specific estimates; Pupil Premium interaction examined. Limitations section emphasizes attrition imbalance and short-term topic learning.","architectureImplications":"Evaluation architecture must version the model/UI/curriculum configuration under study and capture engagement alongside outcomes so estimates remain interpretable as products change.","governanceImplications":"Procurement and research governance should expect iterative micro-evaluations rather than a single multi-year trial before any decision—and should not treat preliminary g≈0.33 as definitive efficacy.","securityPrivacyImplications":"Not a privacy study; still, student assessment and platform engagement data used in micro-RCTs require ordinary education-record protections when districts replicate locally.","caveats":"UK GCSE context; substantial attrition (higher in control); non-standardised curriculum-aligned tests; four-week horizon; arXiv preprint; exploratory engagement analyses are not causal.","streamIds":["k12"],"roles":{"sales":"Use this paper to reset buyer expectations: a positive AI tutoring estimate can coexist with high attrition and short follow-up, so districts should buy evaluation capacity alongside licenses. Position a bounded micro-RCT or rapid A/B as part of the commercial motion, especially when vendors ship frequent model updates. Avoid pitching durable multi-year effect sizes from a four-week topic study.","engineering":"Fit for districts building an evidence loop: instrument baseline/post topic assessments, randomize at student or class level where ethics allow, and freeze a named software version for the window. Proof of value must include attrition diagnostics and confidence intervals, not point estimates alone. Constraints include transferring UK GCSE measures to U.S. standards.","delivery":"Delivery includes recruiting teachers into short randomised windows, managing consent/ethics, minimizing differential attrition, and reporting preliminary results with explicit limits. Ownership should sit with research/assessment plus curriculum. Acceptance means a pre-registered analysis plan and an attrition report; risks include overpublishing preliminary gains into board decisions."},"roleDetails":{"sales":{"customerProblem":"Districts need timely causal signal on fast-changing AI tutors but often rely on usage metrics or slow trials that describe obsolete product versions.","stakeholders":["Research & evaluation lead","Curriculum directors","Procurement","Vendor success / edtech"],"discoveryQuestions":["How often does the vendor change model or tutoring prompts?","What attrition rate would invalidate a four-week classroom pilot for your board?","Do you have curriculum-aligned topic tests ready for rapid pre/post use?"],"valueHypothesis":"Pairing AI tutoring adoption with practitioner-led micro-RCTs can produce timely directional evidence, provided attrition and outcome limits are reported honestly.","potentialEngagement":"Design a four-to-six-week randomised classroom evaluation package with pre-registered ITT analysis and attrition sensitivity checks for one science or math topic.","unsupportedClaims":["Durable year-long learning gains from a four-week study","Zero equity risk based on a single Pupil Premium interaction","Replacement of definitive multi-site trials by one micro-RCT"]},"engineering":{"fit":"AI tutoring platform plus lightweight randomisation and assessment instrumentation for rapid evaluation cycles.","architectureIntegration":"Version-pin tutoring configuration; export assignment arms; collect baseline/post scores and engagement; monitor differential attrition dashboards.","prerequisites":["Ethics/IRB or district research approval","Topic assessments mapped to the intervention","Teacher participation incentives to reduce attrition"],"constraints":["UK GCSE outcome measures may not map 1:1 to U.S. standards","Short horizon captures topic learning, not persistence","Preprint status"],"security":"Secure assessment and platform exports under district data governance; minimize PII in research extracts.","proofOfValue":["ITT effect with confidence interval","Attrition by arm below an agreed threshold or bias analysis if higher","Documented software version under test"]},"delivery":{"work":"Run a short randomised window, chase post-test completion, and publish a preliminary brief with limitations.","dependencies":["Teacher recruitment","Assessment logistics","Vendor cooperation on version freeze"],"ownership":"District research office with curriculum co-owner; vendor supplies configuration freeze.","skillsAdoption":"Teachers need brief protocols for equitable access and post-test completion; analysts need micro-RCT reporting norms.","governanceCheckpoints":["Ethics approval before randomisation","Mid-window attrition check","Results review labeling findings as preliminary before board use"],"acceptanceCriteria":["Pre-registered primary outcome and analysis path","Attrition report by arm","Version identifier recorded for the tutoring stack"],"risks":["Differential attrition biasing estimates","Vendor changing the model mid-trial","Leaders treating preliminary g as definitive efficacy"]}},"retrievedAt":"2026-09-15T18:49:10.771Z","enrichedAt":"2026-09-15T18:49:10.771Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"No clear Pupil Premium differential is not proof of equity; U.S. districts should pre-specify IDEA/ELL subgroup monitoring and accessibility of the tutoring UI in local replications.","procurementImplications":"Write rapid-evaluation and version-freeze obligations into AI tutoring contracts so evidence can keep pace with product changes.","operatingModelImplications":"Shift from one-and-done adoption decisions to a cumulative evaluation operating rhythm owned jointly by research and curriculum.","sourceVerification":{"openedUrl":"https://arxiv.org/pdf/2609.14789","referenceExcerpt":"Attrition was substantial (30.7%), the outcome measures were curriculum-aligned rather","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}