From the K–12 edition of September 15, 2026
UK GCSE science micro-RCT reports preliminary AI tutoring gains while highlighting evaluation lag and attrition
What Worked Education / Durham University (Harrison et al.) · Secondary education research · United Kingdom — English secondary schools (GCSE Biology, Chemistry, Physics)
- Publisher
- arXiv
- Original publication
- 2026-09-13 (arXiv 2609.14789v1)
- Source retrieved
- 2026-09-15
What happened
A four-week multisite individually randomised evaluation of Medly, an AI tutoring platform, in GCSE science found primary ITT Hedges’ g = 0.33 (95% CI 0.18–0.48) versus business-as-usual self-directed revision among students with post-tests (644 of 929 baseline). Positive subject estimates and no clear Pupil Premium differential are reported, but authors label results preliminary because attrition was 30.7%, outcomes were curriculum-aligned rather than standardised, follow-up was short, and process evaluation response was limited. They argue educational AI needs rapid, repeatable micro-RCTs because interventions drift while conventional trials lag.
Why it matters
Transferable scrutiny for U.S. districts: even positive AI tutoring estimates can be fragile under attrition and short horizons, and product version drift can obsolete slow evaluations. Supports building cumulative local randomised tests into edtech adoption—not only one-off pilots.
Evidence and measured results
Individually randomised multisite trial; ITT with Hedges’ g; baseline n=929; post-test n=644; subject-specific estimates; Pupil Premium interaction examined. Limitations section emphasizes attrition imbalance and short-term topic learning.
Limitations and uncertainty
UK GCSE context; substantial attrition (higher in control); non-standardised curriculum-aligned tests; four-week horizon; arXiv preprint; exploratory engagement analyses are not causal.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-15; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Districts need timely causal signal on fast-changing AI tutors but often rely on usage metrics or slow trials that describe obsolete product versions.
- Stakeholders
- Research & evaluation lead
- Curriculum directors
- Procurement
- Vendor success / edtech
- Discovery questions
- How often does the vendor change model or tutoring prompts?
- What attrition rate would invalidate a four-week classroom pilot for your board?
- Do you have curriculum-aligned topic tests ready for rapid pre/post use?
- Value hypothesis
- Pairing AI tutoring adoption with practitioner-led micro-RCTs can produce timely directional evidence, provided attrition and outcome limits are reported honestly.
- Potential engagement
- Design a four-to-six-week randomised classroom evaluation package with pre-registered ITT analysis and attrition sensitivity checks for one science or math topic.
- Unsupported claims
- Durable year-long learning gains from a four-week study
- Zero equity risk based on a single Pupil Premium interaction
- Replacement of definitive multi-site trials by one micro-RCT
Pre-sales engineering
Role takeaway
- Fit
- AI tutoring platform plus lightweight randomisation and assessment instrumentation for rapid evaluation cycles.
- Architecture and integration
- Version-pin tutoring configuration; export assignment arms; collect baseline/post scores and engagement; monitor differential attrition dashboards.
- Prerequisites
- Ethics/IRB or district research approval
- Topic assessments mapped to the intervention
- Teacher participation incentives to reduce attrition
- Constraints
- UK GCSE outcome measures may not map 1:1 to U.S. standards
- Short horizon captures topic learning, not persistence
- Preprint status
- Security
- Secure assessment and platform exports under district data governance; minimize PII in research extracts.
- Proof of value
- ITT effect with confidence interval
- Attrition by arm below an agreed threshold or bias analysis if higher
- Documented software version under test
Delivery
Role takeaway
- Work
- Run a short randomised window, chase post-test completion, and publish a preliminary brief with limitations.
- Dependencies
- Teacher recruitment
- Assessment logistics
- Vendor cooperation on version freeze
- Ownership
- District research office with curriculum co-owner; vendor supplies configuration freeze.
- Skills and adoption
- Teachers need brief protocols for equitable access and post-test completion; analysts need micro-RCT reporting norms.
- Governance checkpoints
- Ethics approval before randomisation
- Mid-window attrition check
- Results review labeling findings as preliminary before board use
- Acceptance criteria
- Pre-registered primary outcome and analysis path
- Attrition report by arm
- Version identifier recorded for the tutoring stack
- Risks
- Differential attrition biasing estimates
- Vendor changing the model mid-trial
- Leaders treating preliminary g as definitive efficacy
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Evaluation architecture must version the model/UI/curriculum configuration under study and capture engagement alongside outcomes so estimates remain interpretable as products change.
Governance
Who approves, reviews and stays accountable for outcomes?
Procurement and research governance should expect iterative micro-evaluations rather than a single multi-year trial before any decision—and should not treat preliminary g≈0.33 as definitive efficacy.
Security and privacy
What data, permissions and controls need testing?
Not a privacy study; still, student assessment and platform engagement data used in micro-RCTs require ordinary education-record protections when districts replicate locally.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
No clear Pupil Premium differential is not proof of equity; U.S. districts should pre-specify IDEA/ELL subgroup monitoring and accessibility of the tutoring UI in local replications.
Procurement
What should contracts, pricing and exit terms secure?
Write rapid-evaluation and version-freeze obligations into AI tutoring contracts so evidence can keep pace with product changes.
Operating model
Which teams own the service once it runs?
Shift from one-and-done adoption decisions to a cumulative evaluation operating rhythm owned jointly by research and curriculum.
Publication history
- 2026-09-15K–12 · Issue 104 resources
Stable resource ID: medly-gcse-micro-rct-20260913