{"resourceId":"thoeni-fryer-rag-tutor-null-osf-2026","versions":[{"version":"external-1564d7fdec9f9da695f0dea4c4cb1772306ece8f1bf15f99ab8c0b12b93e167b","resource":{"id":"thoeni-fryer-rag-tutor-null-osf-2026","title":"Semester-long RAG AI tutor RCT finds null effects on interest, self-efficacy, engagement, and achievement","organization":"University of North Florida & University of Hong Kong (Thoeni & Fryer)","sector":"Higher education teaching and learning / marketing education research","geography":"United States (undergraduate principles of marketing; in-person and asynchronous online)","publishedAt":"2026-04-17","publicationDate":"2026-04-17","eventDate":null,"sourceName":"OSF preprint (journal version in Computers in Human Behavior Reports)","sourceLabel":"Academic randomized controlled field experiment on GenAI tutoring","sourceUrl":"https://osf.io/24tg7/download","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","operating-model","governance-procurement","accessibility-workforce"],"finding":"A semester-long randomized controlled field experiment with about 450 undergraduates across in-person and asynchronous online sections found a course-grounded RAG genAI tutor had no statistically significant impact on interest, self-efficacy, engagement, or academic achievement versus control. Authors urge universities to investigate pedagogical value before substantial investments or long-term contracts. Why now: contrasts with same-week faculty-chatbot success narratives and structured-tutor positives already in the archive, sharpening buy-vs-evaluate discipline for 2026–27 procurement.","sledRelevance":"U.S. public-university null evidence that a validated-content RAG tutor does not automatically move precursor or achievement metrics—critical counterweight when campuses equate AI tutor licenses with student-success gains.","evidence":"Class-level RCT over a 16-week term with pre/post surveys and tests; ~223 treatment; LMS-integrated tutor; OSF project 24tg7 / CHBR article 101061 (Published Apr 17, 2026 per UNF record).","architectureImplications":"Do not assume RAG-on-course-PDF alone is instructional design; instrument usage quality, pair with unaided assessments, and keep human course supports as the control-comparable baseline.","governanceImplications":"Gate multi-year tutor contracts on pre-registered course-level evaluations; publish null and mixed results to faculty governance.","securityPrivacyImplications":"Even null-effect tutors collect student questions inside the LMS tenancy—apply FERPA-aligned retention and access controls regardless of outcome claims.","caveats":"Single marketing course context; tutor quality and content updates may differ from other products; null results do not prove all AI tutors fail—only that this configuration did not move measured outcomes.","streamIds":["student-success"],"roles":{"sales":"For provosts and student-success leaders under pressure to buy campus-wide AI tutors, Thoeni and Fryer’s semester-long U.S. field RCT with about 450 undergraduates finds a RAG genAI tutor produced no statistically significant gains in interest, self-efficacy, engagement, or achievement. Use this cautionary evidence to slow long-term seat commitments until course-level evaluation is funded. Discovery should ask which precursor and grade metrics IR will track and what happens if results are null after one term.","engineering":"Engineering should treat RAG tutors as experimentally unproven for learning unless instrumented like this study: control comparison, pre/post measures, and modality splits (in-person vs async). Architecture implications include LMS-scoped content, usage logs, and easy disablement if null. Constraints: marketing-course evidence; content-update burden; possible underpowered subgroup effects. Security: keep tutor chats in university systems under research/education contracts.","delivery":"Delivery is an evaluation-first pilot: deploy a course-grounded tutor to randomized sections, collect the same precursor and test battery, and pre-commit to a publishable outcome review before scaling licenses. Faculty own content; T&L owns pedagogy; IR owns analysis. Acceptance criteria include documented null-capable metrics and a contract off-ramp. Risks include vendor pressure to expand seats despite null learning signals."},"roleDetails":{"sales":{"customerProblem":"Vendors market always-on RAG tutors as student-success infrastructure without causal learning evidence.","stakeholders":["provost/CAO","teaching & learning","faculty","procurement","IR"],"discoveryQuestions":["Which courses will run a control comparison for one full term?","Which precursor and achievement metrics define success or null?","What contract terms allow exit after a null evaluation?"],"valueHypothesis":"Requiring semester-scale RCTs or quasi-experiments before multi-year tutor buys protects budget and credibility when RAG access alone may not move learning.","potentialEngagement":"One-term evaluated pilot in a high-enrollment course with pre-registered metrics and an explicit scale/no-scale decision.","unsupportedClaims":["That all AI tutors will show null effects","Campus-wide retention guarantees from RAG seats","That engagement analytics equal learning gains"]},"engineering":{"fit":"Fit where LMS integration, section randomization or strong quasi-controls, and IR capacity exist.","architectureIntegration":"Course-scoped RAG corpus in LMS; feature flags by section; export usage and assessment joins to IR.","prerequisites":["Faculty content ownership","Assessment instruments aligned to tutor scope","IRB/local research review if publishing"],"constraints":["Evidence from one marketing course","Tutor efficacy may hinge on unmeasured design details","Null does not identify which UX change would help"],"security":"FERPA-aligned tenancy; limit retention of student prompts; no external model training.","proofOfValue":["Pre-registered contrasts on interest, self-efficacy, engagement, achievement","Modality-stratified reporting","Documented decision rule for scale vs stop"]},"delivery":{"work":"Stand up section-level pilot with control, run full-term measures, and brief academic leadership on null-capable results.","dependencies":["Faculty cooperation","LMS admin support","IR analysis time"],"ownership":"Course lead with T&L co-owner; procurement holds license contingency.","skillsAdoption":"Train faculty not to overclaim tutor benefits in syllabi before evidence.","governanceCheckpoints":["Pre-term evaluation plan approval","Midterm data-quality check","End-term scale/stop decision recorded"],"acceptanceCriteria":["Control and treatment sections measured","Null or effect sizes reported with limitations","No multi-year expansion without review"],"risks":["Political pressure to declare success anyway","Contamination if control students use consumer GenAI","Content drift invalidating the RAG corpus"]}},"retrievedAt":"2026-09-26T03:14:20Z","enrichedAt":"2026-09-26T03:21:39Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Null average effects still require accessible tutor interfaces; do not cut human tutoring based on AI availability when learning gains are unproven.","procurementImplications":"Write evaluation and exit clauses into AI tutor RFPs; prefer term pilots over evergreen seat expansions marketed as retention tools.","operatingModelImplications":"Student-success operating model should fund measurement capacity alongside licenses.","sourceVerification":{"openedUrl":"https://osf.io/24tg7/download","referenceExcerpt":"the genAI tutor had no statistically significant impact on any measured outcome.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}