From the Student Success edition of October 1, 2026
UMD RCT: course-integrated GPT-4o tutor access cuts grades 0.37 SD and LMS participation 0.90 SD; first-gen hurt more
University of Maryland, College Park (Liu, Sweet, Chen, Engelberg, Masters, Clark, Persaud, Lancaster, Hollingsworth, Rice); Annenberg Institute EdWorkingPapers · Higher education undergraduate teaching / AI tutoring · United States (large public research university; University of Maryland)
- Publisher
- EdWorkingPapers (Annenberg Institute at Brown University)
- Original publication
- EdWorkingPaper 26-1598 VERSION October 2026 (manuscript dated September 30, 2026)
- Source retrieved
- 2026-10-02
- Event date
- 2025-08-01
What happened
Instructor-randomized trial of a course-integrated GPT-4o Virtual Study Assistant (VSA) with 2,379 undergraduates and 30 instructors in fall 2025–26. In the exact-match sample (sections of the same course), VSA access reduced final course grades by 0.37 standard deviations (about 4.12 points with controls) and recorded LMS participation by 0.90 SDs, with page views −0.37 SDs and active days −0.51 SDs. Only about 15% of treated students used the VSA at least once (about four sessions). First-generation students (21.9% of sample) showed implied grade effects of −5.10 points (−0.50 SDs) versus −2.23 (−0.22 SDs) for others in the full sample. Authors conclude AI-tutor introduction may displace engagement with existing learning activities without improving performance, with differential impact across groups.
Why it matters
Largest fresh HE RCT signal against assuming course-integrated GenAI tutor access improves grades or engagement. Direct equity flag for first-generation students and a warning that low take-up does not mean null campus impact when instructor practices and ambient AI norms shift.
Evidence and measured results
RCT with instructor-level assignment; full and exact-match samples; administrative grades and ELMS engagement; about 15% take-up. Ambient ChatGPT/Gemini available to both arms; working paper.
Limitations and uncertainty
Instructor-level randomization; ambient AI in both arms; low VSA take-up; single university; working paper not yet peer-reviewed journal version.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-02; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Campuses buy course-integrated GenAI tutors expecting grade and engagement lifts without measuring displacement or first-generation harm.
- Stakeholders
- provost/undergraduate education
- teaching & learning
- academic technology
- student success/equity
- IR/assessment
- Discovery questions
- What is our ITT effect on final grades in matched sections?
- Do LMS participation and page views fall when the tutor launches?
- How do first-generation outcomes compare?
- Value hypothesis
- Measured AI-tutor pilots with displacement and equity endpoints prevent scaling access-only tools that reduce grades and engagement.
- Potential engagement
- One-term multi-section RCT or stepped-wedge pilot of a course-grounded tutor with grade and LMS endpoints plus FGLI heterogeneity.
- Unsupported claims
- Guaranteed GPA improvement from AI tutor seats
- Null impact because take-up is only about 15%
- Equity-neutral deployment without subgroup analysis
Pre-sales engineering
Role takeaway
- Fit
- Course-embedded GenAI virtual study assistants integrated with LMS course materials in large undergraduate courses.
- Architecture and integration
- Instructor-grounded materials → tutor modes → session telemetry → LMS engagement warehouse → equity-flagged outcome dashboards; no auto-grade from chat.
- Prerequisites
- LMS roster and grade export
- instructor activation workflow
- privacy review for chat visibility
- Constraints
- ambient consumer AI available
- low take-up with classroom-level effects
- instructor-level assignment complexity
- Security
- Student–tutor chats as education records; notice if instructors/researchers can view; purpose-limit retention.
- Proof of value
- Non-inferior or improved final grades vs control
- Stable LMS participation/page views
- No widening first-generation grade gap
Delivery
Role takeaway
- Work
- Activate tutor in treated sections, freeze control access, extract grades/ELMS metrics, review weekly displacement, brief equity stakeholders.
- Dependencies
- course instructors
- LMS analytics
- IR for FGLI flags
- IT tutor platform
- Ownership
- Undergraduate course coordinator with academic-technology operations support.
- Skills and adoption
- Instructors redesign activities assuming tutor presence can reduce LMS work; advisors watch FGLI grade trajectories.
- Governance checkpoints
- pre-launch measurement plan
- midterm displacement review
- end-term equity readout
- Acceptance criteria
- Control comparison completed
- Grade and LMS endpoints reported
- First-generation heterogeneity reported
- Risks
- Displacement of course engagement
- First-generation harm
- Misreading low take-up as safety
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Instrument displacement (LMS page views, active days, participation) beside tutor session counts; support instructor-configurable grounding without assuming access equals use; log whether students seek evaluation of their own work versus passive Q&A.
Governance
Who approves, reviews and stays accountable for outcomes?
Require pre-registered grade and engagement analyses—including first-generation heterogeneity—before scaling AI tutors. Disclose ambient AI availability and low take-up when interpreting ITT effects. Do not market seat licenses as learning gains.
Security and privacy
What data, permissions and controls need testing?
Instructor/researcher visibility into student–tutor chats may suppress questions; treat transcripts as education records with purpose limits and notice.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Larger estimated grade losses for first-generation students raise equity risk if AI tutors substitute for structured course engagement that FGLI students rely on. Faculty need capacity to redesign activities when tutors are introduced.
Procurement
What should contracts, pricing and exit terms secure?
Buy measured pilots with displacement and equity endpoints, not unlimited tutor seats. Require exportable engagement/tutor telemetry and ability to disable or redesign when ITT grades fall.
Operating model
Which teams own the service once it runs?
Operate AI tutors as interventions with control comparisons and weekly engagement dashboards; escalate when LMS participation drops or first-gen gaps widen.
Publication history
- 2026-10-01Student Success · Issue 253 resources
Stable resource ID: liu-umd-course-integrated-ai-tutor-rct-202610