Higher Education & ResearchIssue 104 records
Student Success
Four newly archived sources sharpen student-success evidence after the 2026-09-13 edition: a four-year CSUN chatbot RCT finds administrative-task gains with precisely estimated nulls on persistence and GPA; a Middlebury/IZA lab RCT reports 0.27 SD knowledge gains that persist one week, with augmentation uses lasting and automation fading; a July 2026 narrative review maps disability inclusion benefits against algorithmic-exclusion risks; and a Michigan FeedbackWriter RCT improves revision quality without post-test learning gains. One cross-source pattern warns that process metrics are not durable outcomes. Disability-specific causal trials, advising impact, and total cost of ownership remain gaps; several candidate Springer/Taylor & Francis pages were inaccessible.
- Evidence records
- 4
- Cross-source patterns
- 1
- Evidence classes
- 4 academic research
- Outcomes
- 2 cautionary2 mixed
- Source freshness
- 3 recent1 older, newly relevant
- Research completed
- 2026-09-15
Choose a role to see its takeaway beside every record in the ledger.
Patterns across the evidence
Synthesis · Lighthouse Advisory interpretation
Proximal AI process gains often fail to move durable learning or persistence metrics
Across a multi-year student-support chatbot RCT and a large-course AI-mediated feedback RCT, institutions can improve near-term process outcomes—administrative task completion or revision quality—without detectable gains on persistence/GPA or post-test learning. A generative-AI learning RCT similarly shows automation-style quality gains that fade once AI is removed. Interpretation: student-success business cases should pre-specify durable endpoints and treat process metrics as insufficient proof.
Operating questionWhich durable learning, persistence, or unaided-performance endpoints will we pre-register before scaling AI student-success tools that already show process-metric lift?
Supporting evidenceAnnenberg Institute at Brown University / California State University, NorthridgeUniversity of MichiganMiddlebury College / IZA@LISER
Evidence ledger
Full record · every source keeps its link and limitations
Four-year CSUN chatbot RCT finds admin-task gains without persistence or GPA effects
A four-year randomized controlled trial of a non-generative AI text-messaging chatbot at CSUN (N=8,708 across 2018 and 2019 cohorts) finds students remain receptive and complete more time-sensitive administrative tasks, but shows no detectable effects on enrollment/persistence, credits, graduation, or term/cumulative GPA—including among female, Hispanic, Pell-eligible, and first-generation subgroups. Authors emphasize centralized ownership, flexible messaging, and cross-unit data coordination for sustainability.
Why it matters, evidence and limitations
- Why it matters
- Directly informs U.S. public higher-education student-success offices considering AI outreach for retention. Newly relevant after prior student-success editions flagged advising/retention impact and durable outcome gaps: this long-horizon RCT separates administrative responsiveness from persistence claims.
- Evidence and measured results
- Preregistration-style confirmatory framing builds on prior GSU-style evidence; MDES=0.05 SD with large N makes nulls on persistence/GPA informative rather than underpowered. Measured outcomes include enrollment by semester, credits attempted/earned, graduation by year four, and GPA. Implementation evidence comes from system observation and administrator discussions alongside the RCT.
- Limitations and uncertainty
- Working paper (July 2026), not peer-reviewed journal version. Tool is non-generative AI texting—transfer to generative tutoring/advising agents is limited. Subgroup nulls reported as available upon request in text. COVID-era conditions and CSUN context may limit external validity. Spillover cannot be fully ruled out though authors argue design reduces contamination risk.
Lab RCT finds 0.27 SD knowledge gains from GenAI, with augmentation lasting and automation fading
In a proctored randomized experiment with 211 undergraduates learning an unfamiliar topic, generative AI access raised immediate knowledge-test scores by 0.27 SD, with gains persisting about one week later on unaided tests. Essay quality changed little while AI was available but later unaided style/relevance improved more for augmentation users (AI as explainer) than automation users (AI as text generator), whose short-run quality gains faded. Authors caution results are per unit of time in a lab and do not imply campus-wide learning rises if students reallocate saved time.
Why it matters, evidence and limitations
- Why it matters
- Addresses the stream’s durable-learning evidence gap with causal estimates and a usable augmentation-versus-automation distinction for teaching policy. Transferability to large public universities is limited by selective sample and lab setting; still actionable for assessment design and AI literacy.
- Evidence and measured results
- IRB-approved RCT; AI-allowed vs AI-forbidden in Session One; Session Two unaided ~1 week later; usage verified via proctoring, ChatGPT logs, and self-reports; human and AI essay grading plus detector features. Mechanisms include time reallocation toward reading/search and higher reported enjoyment.
- Limitations and uncertainty
- Discussion paper, lab setting, N=211, selective liberal-arts sample, one-week retention only, single technical topic, time-on-task largely fixed by design. External validity to semester courses, high-stakes exams, or equity-priority populations is uncertain. Automation/augmentation labels rely on LLM coding of logs.
Narrative review: HE AI can aid disability access yet risks algorithmic exclusion without UDL governance
This peer-reviewed narrative review of purposive 2020–2026 evidence concludes AI can reduce barriers for students with physical, sensory, cognitive, and learning disabilities when paired with accessible design, institutional support, and participatory governance, but systems trained on non-representative data can replicate or amplify inequities across disability, language, race, income, and digital access. Authors call for rights-based UDL, co-design, transparent governance, and continuous evaluation of access, bias, privacy, and learning outcomes—and warn against replacing human Libras interpreters or denying accommodations because a tool is presumed sufficient.
Why it matters, evidence and limitations
- Why it matters
- Fills the disability-specific evidence gap repeatedly noted in recent student-success editions. U.S. institutions can transfer the governance checklist with limits: Brazilian Libras cases and narrative (non-meta-analytic) methods are not U.S. causal proof, but the risk pattern for assistive vs exclusionary AI is transferable.
- Evidence and measured results
- Narrative synthesis (n=20 items in Table 1) spanning peer-reviewed studies, reviews, policy documents, and Brazil-focused Libras/deaf-inclusion work; explicitly not a PRISMA meta-analysis. Online published 2026-07-18 after June 2026 receipt/acceptance.
- Limitations and uncertainty
- Narrative review without formal risk-of-bias scoring or meta-analysis; heterogeneous evidence; fast-moving tools; authorship includes Brazil-based studies requiring reflexive caution; scarce empirical work on deaf/Libras populations. Not a measured learning or retention trial.
Michigan RCT: TA-mediated AI feedback improves revision quality but not post-test learning
In an IRB-approved RCT in a large UMich undergraduate economics course (354 students, 11 TAs; Jan–Apr 2025), AI suggestions to TAs (FeedbackWriter) produced higher-quality student revisions (Cohen’s d≈0.50) that increased with TA adoption of AI suggestions, and feedback scored as more actionable. Assignment to AI-mediated feedback did not improve post-test scores versus human-only feedback (b=-0.007, p=0.31). Authors attribute limited learning transfer partly to single-practice dosage per knowledge component and call for higher instructional dosage and student-perception studies.
Why it matters, evidence and limitations
- Why it matters
- Shows a practical human-in-the-loop teaching pattern for large courses while reinforcing that revision-quality gains are not automatic learning gains—aligned with student-success emphasis on durable learning over engagement proxies. Newly useful beside retention-null outreach evidence for outcome-definition discipline.
- Evidence and measured results
- Counterbalanced RCT across two writing assignments; multivariate models; automated feedback-quality metrics; TA interviews; 1,366 essays graded with the system per abstract context. Human-only baseline matched UI without AI suggestions.
- Limitations and uncertainty
- Single course/domain; AI evaluator for essay quality; no AI-only comparison arm; optional post-test; limited student voice data; study period 2025 with 2026 CHI/arXiv release—backfill relative to 2026-09-15 but newly paired with retention-null evidence for outcome discipline. Not a retention study.
How to read this edition
Source findings, measured results and limitations come from the cited publications. Patterns, operating questions, role takeaways and implementation considerations are Lighthouse Advisory interpretation, stated as questions to validate locally rather than guaranteed outcomes. Vendor and operator claims are labeled as claims. Full research method.
- Academic research
- Research produced through an academic institution or peer-reviewed venue.