{"resourceId":"liu-umd-course-integrated-ai-tutor-rct-202610","versions":[{"version":"external-cdba245d4f9edcf9915bbc09f2db17a08b2a1ffe3df553ced74128f9340afcf3","resource":{"id":"liu-umd-course-integrated-ai-tutor-rct-202610","title":"UMD RCT: course-integrated GPT-4o tutor access cuts grades 0.37 SD and LMS participation 0.90 SD; first-gen hurt more","organization":"University of Maryland, College Park (Liu, Sweet, Chen, Engelberg, Masters, Clark, Persaud, Lancaster, Hollingsworth, Rice); Annenberg Institute EdWorkingPapers","sector":"Higher education undergraduate teaching / AI tutoring","geography":"United States (large public research university; University of Maryland)","publishedAt":"EdWorkingPaper 26-1598 VERSION October 2026 (manuscript dated September 30, 2026)","publicationDate":"2026-10-01","eventDate":"2025-08-01","sourceName":"EdWorkingPapers (Annenberg Institute at Brown University)","sourceLabel":"Academic working paper (RCT)","sourceUrl":"https://edworkingpapers.com/sites/default/files/ai26-1598.pdf","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","accessibility-workforce","operating-model","governance-procurement"],"finding":"Instructor-randomized trial of a course-integrated GPT-4o Virtual Study Assistant (VSA) with 2,379 undergraduates and 30 instructors in fall 2025–26. In the exact-match sample (sections of the same course), VSA access reduced final course grades by 0.37 standard deviations (about 4.12 points with controls) and recorded LMS participation by 0.90 SDs, with page views −0.37 SDs and active days −0.51 SDs. Only about 15% of treated students used the VSA at least once (about four sessions). First-generation students (21.9% of sample) showed implied grade effects of −5.10 points (−0.50 SDs) versus −2.23 (−0.22 SDs) for others in the full sample. Authors conclude AI-tutor introduction may displace engagement with existing learning activities without improving performance, with differential impact across groups.","sledRelevance":"Largest fresh HE RCT signal against assuming course-integrated GenAI tutor access improves grades or engagement. Direct equity flag for first-generation students and a warning that low take-up does not mean null campus impact when instructor practices and ambient AI norms shift.","evidence":"RCT with instructor-level assignment; full and exact-match samples; administrative grades and ELMS engagement; about 15% take-up. Ambient ChatGPT/Gemini available to both arms; working paper.","architectureImplications":"Instrument displacement (LMS page views, active days, participation) beside tutor session counts; support instructor-configurable grounding without assuming access equals use; log whether students seek evaluation of their own work versus passive Q&A.","governanceImplications":"Require pre-registered grade and engagement analyses—including first-generation heterogeneity—before scaling AI tutors. Disclose ambient AI availability and low take-up when interpreting ITT effects. Do not market seat licenses as learning gains.","securityPrivacyImplications":"Instructor/researcher visibility into student–tutor chats may suppress questions; treat transcripts as education records with purpose limits and notice.","caveats":"Instructor-level randomization; ambient AI in both arms; low VSA take-up; single university; working paper not yet peer-reviewed journal version.","streamIds":["student-success"],"roles":{"sales":"For provosts and teaching-and-learning buyers pitching campus GenAI tutors, this Maryland RCT is a hard caution: access to a course-integrated GPT-4o study assistant cut final grades about 0.37 SD and LMS participation 0.90 SD in matched sections, with first-generation losses more than twice as large, even though only about 15% used the tool. Sell a bounded measurement pilot with displacement and equity endpoints—not seat-count success. Bound claims to this working-paper RCT and refuse guaranteed GPA or retention lifts from tutor access alone.","engineering":"Engineering should instrument LMS page views, active days, and participation next to tutor session telemetry, and preserve instructor grounding controls without assuming access equals productive use. Integrate first-generation flags into outcome dashboards under privacy review. Constraints include ambient ChatGPT/Gemini availability and low observed take-up that still shifted classroom behavior. Proof-of-value is non-inferior grades plus stable LMS engagement in an RCT or strong quasi-experiment—not chat volume. Treat tutor transcripts as FERPA-sensitive with notice when staff can view chats.","delivery":"Delivery should run a course-block pilot with activated versus control sections, weekly displacement reviews, and explicit first-generation outcome monitoring. Ownership sits with the course coordinator and academic-technology lead jointly. Train instructors that introducing a VSA can change study habits even when few students click it. Acceptance means pre-registered grade/engagement analyses and a stop/redesign path if ITT grades fall. Risks include mistaking low take-up for null impact and widening FGLI gaps."},"roleDetails":{"sales":{"customerProblem":"Campuses buy course-integrated GenAI tutors expecting grade and engagement lifts without measuring displacement or first-generation harm.","stakeholders":["provost/undergraduate education","teaching & learning","academic technology","student success/equity","IR/assessment"],"discoveryQuestions":["What is our ITT effect on final grades in matched sections?","Do LMS participation and page views fall when the tutor launches?","How do first-generation outcomes compare?"],"valueHypothesis":"Measured AI-tutor pilots with displacement and equity endpoints prevent scaling access-only tools that reduce grades and engagement.","potentialEngagement":"One-term multi-section RCT or stepped-wedge pilot of a course-grounded tutor with grade and LMS endpoints plus FGLI heterogeneity.","unsupportedClaims":["Guaranteed GPA improvement from AI tutor seats","Null impact because take-up is only about 15%","Equity-neutral deployment without subgroup analysis"]},"engineering":{"fit":"Course-embedded GenAI virtual study assistants integrated with LMS course materials in large undergraduate courses.","architectureIntegration":"Instructor-grounded materials → tutor modes → session telemetry → LMS engagement warehouse → equity-flagged outcome dashboards; no auto-grade from chat.","prerequisites":["LMS roster and grade export","instructor activation workflow","privacy review for chat visibility"],"constraints":["ambient consumer AI available","low take-up with classroom-level effects","instructor-level assignment complexity"],"security":"Student–tutor chats as education records; notice if instructors/researchers can view; purpose-limit retention.","proofOfValue":["Non-inferior or improved final grades vs control","Stable LMS participation/page views","No widening first-generation grade gap"]},"delivery":{"work":"Activate tutor in treated sections, freeze control access, extract grades/ELMS metrics, review weekly displacement, brief equity stakeholders.","dependencies":["course instructors","LMS analytics","IR for FGLI flags","IT tutor platform"],"ownership":"Undergraduate course coordinator with academic-technology operations support.","skillsAdoption":"Instructors redesign activities assuming tutor presence can reduce LMS work; advisors watch FGLI grade trajectories.","governanceCheckpoints":["pre-launch measurement plan","midterm displacement review","end-term equity readout"],"acceptanceCriteria":["Control comparison completed","Grade and LMS endpoints reported","First-generation heterogeneity reported"],"risks":["Displacement of course engagement","First-generation harm","Misreading low take-up as safety"]}},"retrievedAt":"2026-10-02T03:21:35Z","enrichedAt":"2026-10-02T03:28:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Larger estimated grade losses for first-generation students raise equity risk if AI tutors substitute for structured course engagement that FGLI students rely on. Faculty need capacity to redesign activities when tutors are introduced.","procurementImplications":"Buy measured pilots with displacement and equity endpoints, not unlimited tutor seats. Require exportable engagement/tutor telemetry and ability to disable or redesign when ITT grades fall.","operatingModelImplications":"Operate AI tutors as interventions with control comparisons and weekly engagement dashboards; escalate when LMS participation drops or first-gen gaps widen.","sourceVerification":{"openedUrl":"https://edworkingpapers.com/sites/default/files/ai26-1598.pdf","referenceExcerpt":"tutor access reduced final grades by 0.37 standard deviations and learning management system participation by 0.90","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}