{"resourceId":"thoeni-fryer-rag-tutor-null-rct-osf","versions":[{"version":"external-3a00719833df833a8bd842c6b4cd8c4e8a5b23036ddaecee7aa1f6f25a5d78a6","resource":{"id":"thoeni-fryer-rag-tutor-null-rct-osf","title":"Semester-long RAG GenAI tutor RCT finds nulls on interest, self-efficacy, engagement, and tests","organization":"University of North Florida / University of Hong Kong (OSF Preprints)","sector":"Higher education instructional technology / introductory marketing","geography":"United States university (field experiment; multimodal sections)","publishedAt":"OSF Preprint 7 April 2025","publicationDate":"2025-04-07","eventDate":null,"sourceName":"OSF Preprints","sourceLabel":"Thoeni & Fryer — AI Tutors in Higher Education: Comparing Expectations to Evidence","sourceUrl":"https://osf.io/download/uzpn3/","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","operating-model","governance-procurement"],"finding":"Semester-long randomized controlled field experiment (~450 undergraduates; in-person and asynchronous online modalities) of a custom retrieval-augmented generation (RAG) GenAI tutor in introductory marketing found no statistically significant impact on individual interest, self-efficacy, engagement, or course test performance after covariates (gender, age, race, job hours, delivery mode). Students still reported enjoying the tutor and wanting it in other classes, illustrating satisfaction–learning decoupling. Authors urge investigating pedagogical value before large investments or long-term contracts.","sledRelevance":"Newly relevant after recent student-success editions archived CSUN non-generative chatbot nulls and FeedbackWriter revision-without-post-test gains: this GenAI RAG tutor null was not in the archive and supplies Pass B scrutiny on semester-scale tutoring claims. Why now: complements 2026-09-15 durable-outcome gaps with a full-term GenAI tutor achievement null.","evidence":"Source findings: difference-in-differences style tests for treatment×time; Table 2 shows insufficient evidence to reject nulls across interest/self-efficacy/engagement/performance hypotheses; positive student attitude items coexist with null learning metrics. Limitations: single course/instructor (n-of-one); marketing vocabulary-heavy domain; tutor lacked cross-session memory; preprint not journal version (CHBR HTML inaccessible this run).","architectureImplications":"Interpretation: RAG over course content plus chat UI is insufficient without instructional guidance, memory/continuity, and outcome instrumentation beyond satisfaction. Prefer architectures that log learning behaviors and support Socratic/tutor-use guidance rather than open Q&A alone.","governanceImplications":"Interpretation: procurement governance should require pre-registered learning KPIs and exit criteria before multi-year tutor contracts—aligned with authors’ investment caution.","securityPrivacyImplications":"Source emphasizes retrieval of expert course content without training the base model on chats; still involves student prompts. Interpretation: apply FERPA-aware logging, retention, and optional local retrieval stores.","caveats":"Preprint dated April 2025 (older than last run); included for newly relevant GenAI-tutor null not previously archived. Single-course generalizability limited; journal CHBR page could not be opened (404/access error). Satisfaction metrics must not be sold as learning evidence.","streamIds":["student-success"],"roles":{"sales":"Sales framing: challenge vendors equating GenAI tutor adoption or CSAT with learning gains. This semester-long RCT found nulls on interest, self-efficacy, engagement, and tests despite favorable student attitudes. Bound deals to evaluated pilots with learning KPIs and easy exit—authors explicitly warn against substantial investments/long-term contracts on expectation alone.","engineering":"Engineering framing: a content-RAG chatbot without cross-session memory and guided tutoring strategies failed to move measured outcomes. Prerequisites for any PoC: retrieval corpus governance, session memory policy, and instrumentation for learning—not only usage. Constraints: single-course evidence; modality differences as covariates. Proof of value must replicate null-tolerant tests on achievement and precursors.","delivery":"Delivery framing: treat tutor rollout as an instructional experiment with control sections, not an IT feature flip. Dependencies include faculty partnership and assessment design. Ownership: academic department with edtech. Risks: mistaking high satisfaction scores for efficacy; locking into contracts before evidence."},"roleDetails":{"sales":{"customerProblem":"Campuses face pressure to buy always-on GenAI tutors without semester-scale learning evidence.","stakeholders":["CIO/academic technology","Dean of undergraduate studies","Faculty course coordinators","Procurement"],"discoveryQuestions":["What pre-registered learning KPIs define success?","Is satisfaction allowed as a primary success metric?","What is the exit clause if nulls replicate?","Does the tutor retain cross-session student context?"],"valueHypothesis":"A bounded, instrumented pilot can test whether a RAG tutor moves tests/engagement; this evidence predicts nulls without stronger pedagogy.","potentialEngagement":"One-term controlled pilot with explicit null-tolerant evaluation plan and no multi-year lock-in.","unsupportedClaims":["Automatic learning gains from RAG tutors","That student enjoyment equals achievement","Campus-wide ROI from tutor seats alone"]},"engineering":{"fit":"Course-specific RAG tutor for content Q&A; weak fit if the goal is measured learning without pedagogical scaffolding.","architectureIntegration":"RAG over vetted course materials + chat UI; needs session memory, learning analytics, and LMS grade sync for evaluation.","prerequisites":["Curated retrieval corpus","Randomization/assignment mechanism","Pre/post instruments","Usage logs"],"constraints":["Single-course external validity","No cross-session memory in studied tool","Subject-matter fit (vocab-heavy intros)"],"security":"Student prompts may include personal academic struggles; minimize retention and restrict vendor training use.","proofOfValue":["Pre-register DiD on tests and MDL precursors","Power for small effects","Separate satisfaction from learning endpoints"]},"delivery":{"work":"Stand up treated/control sections, train students on tutor use, administer surveys/tests, analyze null-tolerant results.","dependencies":["Instructor partnership","IRB/ethics if research","Stable retrieval content updates"],"ownership":"Course coordinator with academic technology.","skillsAdoption":"Faculty need prompt/corpus stewardship; students need realistic expectations that chat≠mastery.","governanceCheckpoints":["Procurement gate before expansion","Learning-outcomes review after term","Contract exit if nulls"],"acceptanceCriteria":["Completed RCT/pilot report on interest, efficacy, engagement, tests","Documented go/no-go decision"],"risks":["Contract lock-in","Satisfaction washing","Instructor confounding (n-of-one)"]}},"retrievedAt":"2026-09-17T03:08:30Z","enrichedAt":"2026-09-17T03:15:56Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Interpretation: null average effects do not rule out subgroup benefits; authors note equity/access themes in literature but do not report disability-specific impacts—do not infer accessibility wins from this trial.","procurementImplications":"Interpretation: require learning-outcome SLAs, pilot-first purchasing, and cancellation rights; avoid multi-year tutor commitments on CSAT alone.","operatingModelImplications":"Interpretation: operating model should keep human tutoring/faculty pathways primary until GenAI tutors show replicated learning effects.","sourceVerification":{"responseId":"osf-download-uzpn3-2026-09-16","openedUrl":"https://osf.io/download/uzpn3/","referenceExcerpt":"the genAI tutor had no statistically significant impact on any measured outcome.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}