From the Student Success edition of November 10, 2026
UMD working paper (posted Oct 2026): randomized trial of 2,379 undergraduates finds course AI tutor access lowered grades and LMS participation, with larger losses for first-generation students
University of Maryland · Higher education AI tutoring evaluation · United States (Maryland)
- Publisher
- EdWorkingPapers / University of Maryland
- Original publication
- EdWorkingPaper 26-1598, October 2026; UMD College of Education news 16 October 2026
- Source retrieved
- 2026-11-11
What happened
The paper evaluates access to a course-integrated generative AI tutor (a virtual study assistant using retrieval over instructor-selected course materials within the LMS) in a randomized trial of 2,379 undergraduates and 30 instructors in fall 2025. Access reduced final grades by 0.27 to 0.37 standard deviations, with the clearest evidence in the exact-match sample, and reduced course page views (-0.37 SD), active days (-0.51 SD) and recorded participation in instructor-designed activities (-0.90 SD); homework effects were null or negative. Estimated losses were greater among first-generation students. Only about 15% of treated students used the tutor, and users reported asking instructors fewer questions. Authors say the intervention likely captures broader AI use induced by the tutor and list limits including grades as the outcome.
Why it matters
The most rigorous contrary evidence in the archive window to the positive tutor pilots: Duke and Brenau (2 and 6 Nov) report favorable perceptions or exam results without controls, while this randomized trial finds harm from an LMS-integrated tutor. Newly relevant now because those pilots are being used as proof of value.
Evidence and measured results
Randomized trial with within-course random assignment and LMS data, posted as a non-peer-reviewed working paper; effects are on course grades and platform behavior.
Limitations and uncertainty
Working paper; one large public university; grade-based outcomes; low uptake; effects depend on tutor configuration and course context; not peer reviewed.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-11; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Institutions need evidence that an AI tutor helps learning before purchasing or scaling it.
- Stakeholders
- Provost
- CIO
- Teaching and learning center
- Institutional research
- Faculty
- Discovery questions
- What outcome evidence exists beyond usage and satisfaction?
- Can sections be randomized or compared?
- How will equity effects be monitored?
- Value hypothesis
- Interpretation: a pre-deployment evaluation plan may prevent investing in tools that displace learning; the cost-benefit is not measured here.
- Potential engagement
- Tutor evaluation design: comparison sections, independent assessments, engagement and equity dashboards and instructor guidance.
- Unsupported claims
- AI tutors always reduce learning
- AI tutors improve outcomes
- Cross-institution transferability
Pre-sales engineering
Role takeaway
- Fit
- LMS-integrated retrieval-augmented tutor with instructor-controlled materials and modes.
- Architecture and integration
- LMS integration, event logging for page views and participation, instructor configuration of materials, and analytics for subgroup comparison.
- Prerequisites
- LMS event data access
- Comparison or randomized sections
- Independent assessments
- Constraints
- Low uptake (about 15%)
- Grade-based outcomes
- Single university
- Security
- Define log access and retention for student interaction and LMS activity data used in evaluation.
- Proof of value
- Instrument engagement displacement metrics
- Pilot tutor-style versus direct-answer modes
- Compare subgroup outcomes including first-generation students
Delivery
Role takeaway
- Work
- Run staged pilots with measurement, instructor configuration support and student guidance.
- Dependencies
- LMS administrators
- Instructors
- Institutional research
- Student support
- Ownership
- Teaching and learning center with IT and institutional research
- Skills and adoption
- Instructors need guidance to configure modes and to design tasks that keep students engaged with existing activities.
- Governance checkpoints
- Pre-launch evaluation plan approval
- Mid-term engagement and equity review
- Go/no-go on expansion
- Acceptance criteria
- Proposed: no decline in LMS participation versus comparison sections
- Proposed: subgroup outcome gaps tracked and reported
- Proposed: independent assessment results reviewed
- Risks
- Displaced engagement with course activities
- Widening gaps for first-generation students
- Misreading low uptake as success
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
A retrieval-augmented tutor inside the LMS with instructor-selected materials and tutoring or direct-instruction modes; LMS logs were key to detecting displaced participation.
Governance
Who approves, reviews and stays accountable for outcomes?
The authors argue for evaluating AI tools before campus-wide rollout and note low usage and behavior shifts; instructor guidance and task design appear to matter.
Security and privacy
What data, permissions and controls need testing?
Limited discussion in the analyzed excerpts; the study does not report data-security findings.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Larger estimated losses among first-generation students raise equity concerns; students also asked instructors fewer questions and attended fewer office hours.
Procurement
What should contracts, pricing and exit terms secure?
Argues for requiring pre-deployment evaluation and measuring engagement displacement, not just uptake or satisfaction, before scaling a tutor license.
Operating model
Which teams own the service once it runs?
Instructor configuration and course design could change outcomes; the study suggests monitoring instructor-student contact as well as usage.
Publication history
- 2026-11-10Student Success · Issue 645 resources
Stable resource ID: umd-edworkingpaper-course-integrated-ai-tutor-rct-20261016