{"resourceId":"dang-nguyen-expert-verified-ai-study-materials-kcl-20261005","versions":[{"version":"external-664425108929b7e799fe9df903aa67a37cb35d991b0f74a6b25a8e8216785142","resource":{"id":"dang-nguyen-expert-verified-ai-study-materials-kcl-20261005","title":"KCL course study: tutor-verified AI study materials linked to a 24.7-point drop in below-2:1 marks, with gains concentrated among the lowest attainers","organization":"King's Business School, King's College London (Canh Thien Dang, An Nguyen)","sector":"Higher education student success / AI-generated study resources and attainment gaps","geography":"United Kingdom (one compulsory first-year economics course)","publishedAt":"arXiv preprint v1 submitted 5 October 2026 (intervention in 2024/25)","publicationDate":"2026-10-05","eventDate":null,"sourceName":"arXiv (preprint)","sourceLabel":"Academic preprint, quasi-experimental difference-in-differences with qualitative feedback; not yet peer reviewed","sourceUrl":"https://arxiv.org/abs/2610.07097","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["knowledge-work","accessibility-workforce","operating-model","governance-procurement"],"finding":"In 2024/25 the macroeconomics half of a compulsory first-year economics course received NotebookLM-generated podcasts, FAQs and quiz study guides built only from course documents and checked by a named graduate teaching assistant before release under the label 'AI-generated and academically verified'; the microeconomics half and the prior cohort got nothing comparable (two cohorts of 85, 170 students, 340 exam marks). Access was associated with a 2.34-mark advantage on a 50-mark component (p = 0.045 HC1; 0.013 clustered), about 0.33 SD, but the treated half held level (+0.73, n.s.) while the untreated half fell 1.61 marks, so the estimate reflects protection from a decline. Marks below the upper-second (60%) boundary fell 24.7 percentage points (p = 0.001) and below the 50% boundary 12.9 points; effects were significant at every threshold from 23 to 31 marks and none above, and about three-quarters of the average came from the bottom quintile. The average effect loses significance when three to eight of the lowest pre-period students are dropped; the threshold effect does not. Take-up: each podcast reached about 82% of the cohort, FAQs about 65% and guides about 60%. Producing and verifying each weekly set took about 30 minutes (about 2.5 hours across five weeks, roughly 1.8 staff minutes per student). Students (36 gave feedback) read the verification label as a reason to engage without ending their own checking.","sledRelevance":"New this week and directly relevant to universities that now generate AI study materials themselves. It names a concrete control, named expert verification before release, and reports that the benefit landed among the weakest students rather than the strongest. Read with the archived UMD trial: there an open AI tutor coincided with lower grades and engagement, while here students never prompted a model and a tutor carried the checking.","evidence":"Two-cohort difference-in-differences within one course (treated vs untreated half, before vs after), HC1, student-clustered and within-student specifications; threshold linear probability models, quantile and variance tests; robustness by dropping or winsorising low scorers; VLE take-up counts; field notes from 36 students. One pre-intervention cohort, so parallel trends cannot be tested; individual usage logs were not retained.","architectureImplications":"The deployment used a source-grounded generator restricted to uploaded course documents under institutional terms excluding training use, plus a human verification gate. The design pattern is generator, verifier, labelled release, rather than an open student-facing chatbot.","governanceImplications":"Interpretation: course-issued AI material should carry a named accountable verifier and a disclosure label, with a log of verification time and changes. The authors warn that verification work should be counted in workload, not absorbed into insecure graduate-assistant contracts.","securityPrivacyImplications":"Inputs were course slides, readings and assessment guidance, not student data; the institution's arrangement excluded uploaded content from model training. Extending the pattern to student-specific materials would need a separate privacy review.","caveats":"Single course, single UK institution, one pre-period cohort; cohort differences and effort substitution between halves cannot be excluded; average effect rests on a few low-scoring students; verification was bundled with new material in three formats, so its separate contribution is unidentified; high-attaining cohort with no failures; time costs are one verifier's estimates without an audit; preprint, not peer reviewed.","streamIds":["student-success"],"roles":{"sales":"The framing point is accountability, not generation. This UK study ties tutor-verified AI study materials to fewer below-2:1 marks, with most of the benefit among the lowest attainers. Use it to open conversations about governed, course-issued AI content with a named verifier and disclosure label. Keep it bounded: one course, a quasi-experiment, and an average effect that weakens when a few low scorers are removed. It supports a pilot with distributional measurement, not a claim of improved grades.","engineering":"For engineering, the framing point is a gated pipeline: a source-grounded generator limited to course documents, a human verification step, and labelled release through the LMS. Check that the generator cannot draw on outside content, that institutional terms exclude training on uploads, and that each item records who verified it and what changed. Usage logs were not kept in the study, so instrument item-level use from the start so outcomes can be linked to use by attainment band.","delivery":"Delivery's framing point is that verification is real work that needs an owner and time. Budget about 30 minutes per weekly set as a starting estimate and confirm it locally with a time log. Assign a named subject-expert verifier, keep a change log, and publish materials with an AI-and-verified label. Plan evaluation that reports threshold and lower-tail results, and if possible randomise verified versus unverified releases, which the study itself could not do."},"roleDetails":{"sales":{"customerProblem":"Course teams can now generate study materials with AI cheaply, but unverified output shifts the checking burden to students least able to carry it.","stakeholders":["Provost or dean of teaching and learning","Teaching and learning center","Department heads of large gateway courses","Academic technology","Institutional research"],"discoveryQuestions":["Are instructors already issuing AI-generated study materials, and who checks them?","Which gateway courses have large shares of students near a grade boundary?","Can you log use of course resources at the item level?"],"valueHypothesis":"Tutor-verified, source-grounded study materials may reduce the share of students below key grade boundaries in gateway courses; the effect must be confirmed locally.","potentialEngagement":"A one-term pilot in two or three gateway courses with a verification workflow, labelled release and distributional evaluation.","unsupportedClaims":["That AI study materials raise average grades","That the effect holds in courses with high failure rates","That verification is free or can be skipped"]},"engineering":{"fit":"Institutions with an LMS and a source-grounded generation tool under no-training terms.","architectureIntegration":"Source-grounded generator fed course documents, verification queue with reviewer sign-off and change log, labelled publish to the LMS, and item-level usage telemetry to the IR warehouse.","prerequisites":["Institutional terms excluding uploaded content from model training","LMS item-level usage logging","Named verifier per course"],"constraints":["Single UK course evidence base","Source-grounding reduces but does not eliminate errors","Copyright and licensing of uploaded readings"],"security":"Limit inputs to course materials, keep student data out of the generator, and retain verification records for audit.","proofOfValue":["Compare below-boundary shares between verified-material and comparison sections","Link item-level use to outcomes by prior attainment","Audit verifier time and error-correction rate per item"]},"delivery":{"work":"Stand up a generate-verify-label workflow for selected gateway courses and evaluate outcomes by attainment band.","dependencies":["Instructor provision of source documents","Verifier time allocation","IR support for distributional analysis"],"ownership":"Course leader owns content; a named subject-expert verifier owns release; teaching and learning center owns the workflow.","skillsAdoption":"Train verifiers on fidelity checks against sources and on logging corrections; brief students on what the label does and does not guarantee.","governanceCheckpoints":["Copyright and terms review before upload","Verifier sign-off on every item","End-of-term review of verification logs and outcomes"],"acceptanceCriteria":["Every released item carries a named verifier and disclosure label","Verification time logged per item","Below-boundary share reported against a comparison group"],"risks":["Verification quietly cut to save time","Effort shifting from other course parts","Over-reading a fragile average effect"]}},"retrievedAt":"2026-10-07T03:12:18Z","enrichedAt":"2026-10-07T03:17:13Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Gains concentrated in the bottom quintile suggest verified, multi-format materials may help weaker students; the verifier role is skilled academic labour that needs time allocation.","procurementImplications":"Value depends on a source-grounded tool with no-training terms plus staff verification time; licences alone do not reproduce the design.","operatingModelImplications":"Assign verification to subject-expert staff, budget roughly 30 minutes per weekly set, label outputs, and report effects by attainment band rather than averages only.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.07097","referenceExcerpt":"The share of marks below the upper-second classification boundary fell by 24.7 percentage points relative to the counterfactual","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}