{"resourceId":"kreijkes-llm-notes-secondary-rct-2026","versions":[{"version":"external-3274e6ceac3d58b6171cf302d4ad39231832b36191c8bc761a420a3b2b266055","resource":{"id":"kreijkes-llm-notes-secondary-rct-2026","title":"Secondary-school RCT: note-taking beats LLM-alone for reading retention despite student preference for AI","organization":"Cambridge University Press & Assessment / Microsoft Research (Computers & Education)","sector":"Academic research / secondary education","geography":"England, United Kingdom (transferable caution for U.S. K-12)","publishedAt":"2026 (Computers & Education 243); available online 24 November 2025","publicationDate":"2026-01-01","eventDate":null,"sourceName":"Computers & Education (Kreijkes et al.)","sourceLabel":"Peer-reviewed randomized experiment","sourceUrl":"https://researchdmr.com/files/LLM_Notes_Reading_Comp.pdf","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","operating-model","accessibility-workforce"],"finding":"Why now as Florida and other systems still permit instructor-authorized AI tutoring/feedback: a pre-registered England secondary RCT (n=405, ages 14–15) finds note-taking alone and LLM+notes significantly outperform LLM-alone on reading comprehension and three-day retention, even though most students prefer the LLM and rate it more helpful—showing preference and cognitive-load reduction are not reliable proxies for independent learning.","sledRelevance":"U.S. districts designing allowed AI reading supports should require effortful encoding activities and off-tool assessments; transferability is limited by England sample and short retention window but directionally relevant to cognitive-offloading concerns.","evidence":"Kreijkes et al., Computers & Education 243 (2026) 105514, DOI 10.1016/j.compedu.2025.105514. Pre-registered RCT with within- and between-participant elements in English secondary schools; 405 students studied two passages; outcomes literal retention, comprehension, free recall after three days; mixed-effects models; qualitative preference/archetype analyses. Abstract and results state significant positive effects of notes and LLM+notes vs LLM alone; preference mismatch documented.","architectureImplications":"If deploying LLM reading aids, default workflows should pair AI explanation with mandatory note-taking or other generative learning activities and block answer-only prompting patterns identified as weak archetypes.","governanceImplications":"Allowed-use policies should distinguish scaffolding from substitution; student satisfaction metrics cannot alone greenlight expansion.","securityPrivacyImplications":"School-managed LLM sessions still need age-appropriate logging and prompt controls; study itself is learning-focused rather than privacy-focused.","caveats":"England sample; three-day retention; Microsoft Research co-authorship; journal volume dated 2026 with online availability Nov 2025—older than last SLED run but newly relevant to current U.S. permitted-use debates; not a U.S. district field trial.","streamIds":["k12"],"roles":{"sales":"Sales framing: do not cite student preference or feels-helpful scores as efficacy evidence. This secondary-school RCT shows LLM-alone underperformed traditional note-taking on comprehension and three-day retention even though learners preferred the model. Sell instructional designs that combine AI with effortful encoding, and set clear expectations for off-tool assessment in any reading or tutoring pilot.","engineering":"Engineering framing: product defaults should discourage pure answer-seeking during reading support. Prompt scaffolds, mandatory note-capture, and post-session closed-book checks belong in the learning path rather than as optional teacher reminders. Instrument prompting archetypes so schools can see substitution versus elaboration behaviors and intervene when students offload encoding work to the model.","delivery":"Delivery framing: pilots must pre-register off-tool retention and comprehension measures and compare AI-alone versus AI-plus-notes conditions under teacher supervision. Train teachers to require generative notes and to interpret preference survey spikes as engagement signals, not mastery. Scale only when delayed learning measures clear a pre-agreed bar against the notes baseline."},"roleDetails":{"sales":{"customerProblem":"Districts authorizing AI reading supports lack evidence that student preference translates into retained learning.","stakeholders":["Curriculum directors","Secondary principals","Instructional coaches","Assessment leads"],"discoveryQuestions":["Will pilots measure closed-book retention after several days?","Are students required to take notes while using the LLM?","What prompting behaviors will you disallow?","How will you separate preference surveys from mastery evidence?"],"valueHypothesis":"A bounded reading-support pilot that mandates notes-plus-LLM and off-tool tests can demonstrate responsible use without claiming LLM-alone superiority.","potentialEngagement":"6–8 week secondary reading pilot design: conditions, measures, teacher protocols, and go/no-go criteria tied to retention—not NPS.","unsupportedClaims":["That LLM use improves long-term learning in this study","Direct U.S. causal generalization without local replication","Guaranteed grade gains from chatbot access"]},"engineering":{"fit":"Fit for supervised secondary reading tools that can enforce note capture and disable answer-only modes; poor fit for unsupervised homework chatbots.","architectureIntegration":"Session design: passage → constrained LLM help → mandatory notes → delayed closed-book quiz; analytics on prompt archetypes.","prerequisites":["Teacher-supervised sessions","Assessment items for delayed tests","Logging of prompt types"],"constraints":["England adolescent sample","Short retention interval","May not match proprietary U.S. tutor pedagogies"],"security":"Keep passage and student responses in district-controlled environments; minimize retention of prompts beyond evaluation needs.","proofOfValue":["Off-tool retention not worse than notes baseline","Reduced pure answer-seeking prompt share","Teacher-confirmed note artifacts per session"]},"delivery":{"work":"Co-design pilot protocols, train teachers on note requirements, run delayed assessments, and report preference vs learning separately.","dependencies":["English/literacy department","Assessment office","Scheduling for two-session design"],"ownership":"Secondary curriculum lead owns instructional protocol; research/assessment owns measurement.","skillsAdoption":"Teachers need PD on prompting archetypes and on resisting student pressure to drop notes because AI 'feels easier.'","governanceCheckpoints":["Pre-registration of measures","Mid-pilot fidelity check on note requirements","Results review before scaling"],"acceptanceCriteria":["Delayed comprehension/retention analyzed by condition","Preference data reported separately from learning","Written decision on scale/stop/modify"],"risks":["Students abandon notes when unsupervised","Over-generalizing UK results to U.S. elementary","Vendor dashboards that optimize engagement over retention"]}},"retrievedAt":"2026-09-18T03:18:59Z","enrichedAt":"2026-09-18T03:24:14Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"LLM accessibility benefits (simpler language) can trade off against memory encoding; teachers need strategies that preserve access without removing generative learning effort.","procurementImplications":"Require vendors to support notes-plus-AI workflows and independent delayed assessments in pilots; reject preference-only success metrics.","operatingModelImplications":"Shift from access rollout KPIs to instructional-condition KPIs (AI-alone vs AI+notes) before expanding secondary reading AI.","sourceVerification":{"openedUrl":"https://researchdmr.com/files/LLM_Notes_Reading_Comp.pdf","referenceExcerpt":"both note-taking alone and combined with LLM use had significant positive effects on retention and comprehension compared to using the LLM alone","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}