{"resourceId":"umd-edworkingpaper-course-integrated-ai-tutor-rct-20261016","versions":[{"version":"external-9ec353b069dfe9e2670293523d4dcc28301bc48bcfaa0b495ab8bdaf578a73e3","resource":{"id":"umd-edworkingpaper-course-integrated-ai-tutor-rct-20261016","title":"UMD working paper (posted Oct 2026): randomized trial of 2,379 undergraduates finds course AI tutor access lowered grades and LMS participation, with larger losses for first-generation students","organization":"University of Maryland","sector":"Higher education AI tutoring evaluation","geography":"United States (Maryland)","publishedAt":"EdWorkingPaper 26-1598, October 2026; UMD College of Education news 16 October 2026","publicationDate":"2026-10-16","eventDate":null,"sourceName":"EdWorkingPapers / University of Maryland","sourceLabel":"Randomized controlled trial working paper","sourceUrl":"https://edsi.umd.edu/sites/default/files/ai26-1598.pdf","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","accessibility-workforce","operating-model"],"finding":"The paper evaluates access to a course-integrated generative AI tutor (a virtual study assistant using retrieval over instructor-selected course materials within the LMS) in a randomized trial of 2,379 undergraduates and 30 instructors in fall 2025. Access reduced final grades by 0.27 to 0.37 standard deviations, with the clearest evidence in the exact-match sample, and reduced course page views (-0.37 SD), active days (-0.51 SD) and recorded participation in instructor-designed activities (-0.90 SD); homework effects were null or negative. Estimated losses were greater among first-generation students. Only about 15% of treated students used the tutor, and users reported asking instructors fewer questions. Authors say the intervention likely captures broader AI use induced by the tutor and list limits including grades as the outcome.","sledRelevance":"The most rigorous contrary evidence in the archive window to the positive tutor pilots: Duke and Brenau (2 and 6 Nov) report favorable perceptions or exam results without controls, while this randomized trial finds harm from an LMS-integrated tutor. Newly relevant now because those pilots are being used as proof of value.","evidence":"Randomized trial with within-course random assignment and LMS data, posted as a non-peer-reviewed working paper; effects are on course grades and platform behavior.","architectureImplications":"A retrieval-augmented tutor inside the LMS with instructor-selected materials and tutoring or direct-instruction modes; LMS logs were key to detecting displaced participation.","governanceImplications":"The authors argue for evaluating AI tools before campus-wide rollout and note low usage and behavior shifts; instructor guidance and task design appear to matter.","securityPrivacyImplications":"Limited discussion in the analyzed excerpts; the study does not report data-security findings.","caveats":"Working paper; one large public university; grade-based outcomes; low uptake; effects depend on tutor configuration and course context; not peer reviewed.","streamIds":["student-success"],"roles":{"sales":"Sales framing: buyers hearing positive tutor pilots now have a large randomized counterexample. This UMD trial of 2,379 students found course-AI access lowered grades and LMS participation, with larger losses for first-generation students. The credible engagement is an evaluation design and guardrail package before rollout, not a tutor license pitch. It is a working paper on one configuration at one university, so avoid either blanket claims that AI tutors harm or help.","engineering":"Engineering framing: fit is retrieval-augmented tutors inside the LMS, with logs that reveal displaced engagement. Prerequisites are LMS event data, per-section randomization or comparison groups, and independent assessments rather than course grades alone. Constraints include low uptake and tool configuration. Validate by instrumenting page views, active days and instructor contact, and testing modes that guide tutor-style use over direct answers.","delivery":"Delivery framing: delivery should treat the tutor as an intervention needing measurement. Owners are the teaching center and IT, with instructors configuring modes and materials. Run a staged pilot with comparison sections, monitor engagement and equity by student group, and give students guidance on tutor use. Acceptance should be proposed thresholds on engagement and assessment performance, since this evidence shows harm is possible and mechanisms are unproven."},"roleDetails":{"sales":{"customerProblem":"Institutions need evidence that an AI tutor helps learning before purchasing or scaling it.","stakeholders":["Provost","CIO","Teaching and learning center","Institutional research","Faculty"],"discoveryQuestions":["What outcome evidence exists beyond usage and satisfaction?","Can sections be randomized or compared?","How will equity effects be monitored?"],"valueHypothesis":"Interpretation: a pre-deployment evaluation plan may prevent investing in tools that displace learning; the cost-benefit is not measured here.","potentialEngagement":"Tutor evaluation design: comparison sections, independent assessments, engagement and equity dashboards and instructor guidance.","unsupportedClaims":["AI tutors always reduce learning","AI tutors improve outcomes","Cross-institution transferability"]},"engineering":{"fit":"LMS-integrated retrieval-augmented tutor with instructor-controlled materials and modes.","architectureIntegration":"LMS integration, event logging for page views and participation, instructor configuration of materials, and analytics for subgroup comparison.","prerequisites":["LMS event data access","Comparison or randomized sections","Independent assessments"],"constraints":["Low uptake (about 15%)","Grade-based outcomes","Single university"],"security":"Define log access and retention for student interaction and LMS activity data used in evaluation.","proofOfValue":["Instrument engagement displacement metrics","Pilot tutor-style versus direct-answer modes","Compare subgroup outcomes including first-generation students"]},"delivery":{"work":"Run staged pilots with measurement, instructor configuration support and student guidance.","dependencies":["LMS administrators","Instructors","Institutional research","Student support"],"ownership":"Teaching and learning center with IT and institutional research","skillsAdoption":"Instructors need guidance to configure modes and to design tasks that keep students engaged with existing activities.","governanceCheckpoints":["Pre-launch evaluation plan approval","Mid-term engagement and equity review","Go/no-go on expansion"],"acceptanceCriteria":["Proposed: no decline in LMS participation versus comparison sections","Proposed: subgroup outcome gaps tracked and reported","Proposed: independent assessment results reviewed"],"risks":["Displaced engagement with course activities","Widening gaps for first-generation students","Misreading low uptake as success"]}},"retrievedAt":"2026-11-11T04:16:00Z","enrichedAt":"2026-11-11T04:19:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Larger estimated losses among first-generation students raise equity concerns; students also asked instructors fewer questions and attended fewer office hours.","procurementImplications":"Argues for requiring pre-deployment evaluation and measuring engagement displacement, not just uptake or satisfaction, before scaling a tutor license.","operatingModelImplications":"Instructor configuration and course design could change outcomes; the study suggests monitoring instructor-student contact as well as usage.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://edsi.umd.edu/sites/default/files/ai26-1598.pdf","referenceExcerpt":"tutor access reduced final grades by 0.37 standard deviations and learning management system participation by 0.90 standard deviations.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}