From the Student Success edition of September 16, 2026
Semester-long RAG GenAI tutor RCT finds nulls on interest, self-efficacy, engagement, and tests
University of North Florida / University of Hong Kong (OSF Preprints) · Higher education instructional technology / introductory marketing · United States university (field experiment; multimodal sections)
- Publisher
- OSF Preprints
- Original publication
- OSF Preprint 7 April 2025
- Source retrieved
- 2026-09-17
What happened
Semester-long randomized controlled field experiment (~450 undergraduates; in-person and asynchronous online modalities) of a custom retrieval-augmented generation (RAG) GenAI tutor in introductory marketing found no statistically significant impact on individual interest, self-efficacy, engagement, or course test performance after covariates (gender, age, race, job hours, delivery mode). Students still reported enjoying the tutor and wanting it in other classes, illustrating satisfaction–learning decoupling. Authors urge investigating pedagogical value before large investments or long-term contracts.
Why it matters
Newly relevant after recent student-success editions archived CSUN non-generative chatbot nulls and FeedbackWriter revision-without-post-test gains: this GenAI RAG tutor null was not in the archive and supplies Pass B scrutiny on semester-scale tutoring claims. Why now: complements 2026-09-15 durable-outcome gaps with a full-term GenAI tutor achievement null.
Evidence and measured results
Source findings: difference-in-differences style tests for treatment×time; Table 2 shows insufficient evidence to reject nulls across interest/self-efficacy/engagement/performance hypotheses; positive student attitude items coexist with null learning metrics. Limitations: single course/instructor (n-of-one); marketing vocabulary-heavy domain; tutor lacked cross-session memory; preprint not journal version (CHBR HTML inaccessible this run).
Limitations and uncertainty
Preprint dated April 2025 (older than last run); included for newly relevant GenAI-tutor null not previously archived. Single-course generalizability limited; journal CHBR page could not be opened (404/access error). Satisfaction metrics must not be sold as learning evidence.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-17; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Campuses face pressure to buy always-on GenAI tutors without semester-scale learning evidence.
- Stakeholders
- CIO/academic technology
- Dean of undergraduate studies
- Faculty course coordinators
- Procurement
- Discovery questions
- What pre-registered learning KPIs define success?
- Is satisfaction allowed as a primary success metric?
- What is the exit clause if nulls replicate?
- Does the tutor retain cross-session student context?
- Value hypothesis
- A bounded, instrumented pilot can test whether a RAG tutor moves tests/engagement; this evidence predicts nulls without stronger pedagogy.
- Potential engagement
- One-term controlled pilot with explicit null-tolerant evaluation plan and no multi-year lock-in.
- Unsupported claims
- Automatic learning gains from RAG tutors
- That student enjoyment equals achievement
- Campus-wide ROI from tutor seats alone
Pre-sales engineering
Role takeaway
- Fit
- Course-specific RAG tutor for content Q&A; weak fit if the goal is measured learning without pedagogical scaffolding.
- Architecture and integration
- RAG over vetted course materials + chat UI; needs session memory, learning analytics, and LMS grade sync for evaluation.
- Prerequisites
- Curated retrieval corpus
- Randomization/assignment mechanism
- Pre/post instruments
- Usage logs
- Constraints
- Single-course external validity
- No cross-session memory in studied tool
- Subject-matter fit (vocab-heavy intros)
- Security
- Student prompts may include personal academic struggles; minimize retention and restrict vendor training use.
- Proof of value
- Pre-register DiD on tests and MDL precursors
- Power for small effects
- Separate satisfaction from learning endpoints
Delivery
Role takeaway
- Work
- Stand up treated/control sections, train students on tutor use, administer surveys/tests, analyze null-tolerant results.
- Dependencies
- Instructor partnership
- IRB/ethics if research
- Stable retrieval content updates
- Ownership
- Course coordinator with academic technology.
- Skills and adoption
- Faculty need prompt/corpus stewardship; students need realistic expectations that chat≠mastery.
- Governance checkpoints
- Procurement gate before expansion
- Learning-outcomes review after term
- Contract exit if nulls
- Acceptance criteria
- Completed RCT/pilot report on interest, efficacy, engagement, tests
- Documented go/no-go decision
- Risks
- Contract lock-in
- Satisfaction washing
- Instructor confounding (n-of-one)
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
RAG over course content plus chat UI is insufficient without instructional guidance, memory/continuity, and outcome instrumentation beyond satisfaction. Prefer architectures that log learning behaviors and support Socratic/tutor-use guidance rather than open Q&A alone.
Governance
Who approves, reviews and stays accountable for outcomes?
Procurement governance should require pre-registered learning KPIs and exit criteria before multi-year tutor contracts—aligned with authors’ investment caution.
Security and privacy
What data, permissions and controls need testing?
Source emphasizes retrieval of expert course content without training the base model on chats; still involves student prompts. Interpretation: apply FERPA-aware logging, retention, and optional local retrieval stores.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Null average effects do not rule out subgroup benefits; authors note equity/access themes in literature but do not report disability-specific impacts—do not infer accessibility wins from this trial.
Procurement
What should contracts, pricing and exit terms secure?
Require learning-outcome SLAs, pilot-first purchasing, and cancellation rights; avoid multi-year tutor commitments on CSAT alone.
Operating model
Which teams own the service once it runs?
Operating model should keep human tutoring/faculty pathways primary until GenAI tutors show replicated learning effects.
Publication history
- 2026-09-16Student Success · Issue 115 resources
Stable resource ID: thoeni-fryer-rag-tutor-null-rct-osf