From the Student Success edition of September 25, 2026
Semester-long RAG AI tutor RCT finds null effects on interest, self-efficacy, engagement, and achievement
University of North Florida & University of Hong Kong (Thoeni & Fryer) · Higher education teaching and learning / marketing education research · United States (undergraduate principles of marketing; in-person and asynchronous online)
- Publisher
- OSF preprint (journal version in Computers in Human Behavior Reports)
- Original publication
- 2026-04-17
- Source retrieved
- 2026-09-26
What happened
A semester-long randomized controlled field experiment with about 450 undergraduates across in-person and asynchronous online sections found a course-grounded RAG genAI tutor had no statistically significant impact on interest, self-efficacy, engagement, or academic achievement versus control. Authors urge universities to investigate pedagogical value before substantial investments or long-term contracts. Why now: contrasts with same-week faculty-chatbot success narratives and structured-tutor positives already in the archive, sharpening buy-vs-evaluate discipline for 2026–27 procurement.
Why it matters
U.S. public-university null evidence that a validated-content RAG tutor does not automatically move precursor or achievement metrics—critical counterweight when campuses equate AI tutor licenses with student-success gains.
Evidence and measured results
Class-level RCT over a 16-week term with pre/post surveys and tests; ~223 treatment; LMS-integrated tutor; OSF project 24tg7 / CHBR article 101061 (Published Apr 17, 2026 per UNF record).
Limitations and uncertainty
Single marketing course context; tutor quality and content updates may differ from other products; null results do not prove all AI tutors fail—only that this configuration did not move measured outcomes.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-26; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Vendors market always-on RAG tutors as student-success infrastructure without causal learning evidence.
- Stakeholders
- provost/CAO
- teaching & learning
- faculty
- procurement
- IR
- Discovery questions
- Which courses will run a control comparison for one full term?
- Which precursor and achievement metrics define success or null?
- What contract terms allow exit after a null evaluation?
- Value hypothesis
- Requiring semester-scale RCTs or quasi-experiments before multi-year tutor buys protects budget and credibility when RAG access alone may not move learning.
- Potential engagement
- One-term evaluated pilot in a high-enrollment course with pre-registered metrics and an explicit scale/no-scale decision.
- Unsupported claims
- That all AI tutors will show null effects
- Campus-wide retention guarantees from RAG seats
- That engagement analytics equal learning gains
Pre-sales engineering
Role takeaway
- Fit
- Fit where LMS integration, section randomization or strong quasi-controls, and IR capacity exist.
- Architecture and integration
- Course-scoped RAG corpus in LMS; feature flags by section; export usage and assessment joins to IR.
- Prerequisites
- Faculty content ownership
- Assessment instruments aligned to tutor scope
- IRB/local research review if publishing
- Constraints
- Evidence from one marketing course
- Tutor efficacy may hinge on unmeasured design details
- Null does not identify which UX change would help
- Security
- FERPA-aligned tenancy; limit retention of student prompts; no external model training.
- Proof of value
- Pre-registered contrasts on interest, self-efficacy, engagement, achievement
- Modality-stratified reporting
- Documented decision rule for scale vs stop
Delivery
Role takeaway
- Work
- Stand up section-level pilot with control, run full-term measures, and brief academic leadership on null-capable results.
- Dependencies
- Faculty cooperation
- LMS admin support
- IR analysis time
- Ownership
- Course lead with T&L co-owner; procurement holds license contingency.
- Skills and adoption
- Train faculty not to overclaim tutor benefits in syllabi before evidence.
- Governance checkpoints
- Pre-term evaluation plan approval
- Midterm data-quality check
- End-term scale/stop decision recorded
- Acceptance criteria
- Control and treatment sections measured
- Null or effect sizes reported with limitations
- No multi-year expansion without review
- Risks
- Political pressure to declare success anyway
- Contamination if control students use consumer GenAI
- Content drift invalidating the RAG corpus
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Do not assume RAG-on-course-PDF alone is instructional design; instrument usage quality, pair with unaided assessments, and keep human course supports as the control-comparable baseline.
Governance
Who approves, reviews and stays accountable for outcomes?
Gate multi-year tutor contracts on pre-registered course-level evaluations; publish null and mixed results to faculty governance.
Security and privacy
What data, permissions and controls need testing?
Even null-effect tutors collect student questions inside the LMS tenancy—apply FERPA-aligned retention and access controls regardless of outcome claims.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Null average effects still require accessible tutor interfaces; do not cut human tutoring based on AI availability when learning gains are unproven.
Procurement
What should contracts, pricing and exit terms secure?
Write evaluation and exit clauses into AI tutor RFPs; prefer term pilots over evergreen seat expansions marketed as retention tools.
Operating model
Which teams own the service once it runs?
Student-success operating model should fund measurement capacity alongside licenses.
Publication history
- 2026-09-25Student Success · Issue 204 resources
Stable resource ID: thoeni-fryer-rag-tutor-null-osf-2026