{"resourceId":"csun-ai-chatbot-four-year-rct-2026","versions":[{"version":"external-f03c030abc5d6a6f011d775b95de47d80cb4e8a571539b4b0f7bb39a77144ad9","resource":{"id":"csun-ai-chatbot-four-year-rct-2026","title":"Four-year CSUN chatbot RCT finds admin-task gains without persistence or GPA effects","organization":"Annenberg Institute at Brown University / California State University, Northridge","sector":"Higher education student success / public university","geography":"United States (California State University, Northridge)","publishedAt":"July 2026","publicationDate":"2026-07-01","eventDate":null,"sourceName":"EdWorkingPapers","sourceLabel":"EdWorkingPaper No. 26-1409 (Mata, Russell & Page)","sourceUrl":"https://edworkingpapers.com/sites/default/files/ai26-1409_v2.pdf","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["operating-model","accessibility-workforce","governance-procurement"],"finding":"A four-year randomized controlled trial of a non-generative AI text-messaging chatbot at CSUN (N=8,708 across 2018 and 2019 cohorts) finds students remain receptive and complete more time-sensitive administrative tasks, but shows no detectable effects on enrollment/persistence, credits, graduation, or term/cumulative GPA—including among female, Hispanic, Pell-eligible, and first-generation subgroups. Authors emphasize centralized ownership, flexible messaging, and cross-unit data coordination for sustainability.","sledRelevance":"Directly informs U.S. public higher-education student-success offices considering AI outreach for retention. Newly relevant after prior student-success editions flagged advising/retention impact and durable outcome gaps: this long-horizon RCT separates administrative responsiveness from persistence claims.","evidence":"Preregistration-style confirmatory framing builds on prior GSU-style evidence; MDES=0.05 SD with large N makes nulls on persistence/GPA informative rather than underpowered. Measured outcomes include enrollment by semester, credits attempted/earned, graduation by year four, and GPA. Implementation evidence comes from system observation and administrator discussions alongside the RCT.","architectureImplications":"Non-generative, centrally owned messaging integrated with registrar/financial-aid administrative data; sustainability depends on shared student-level data feeds and campaign orchestration across units rather than a standalone chatbot silo. Interpretation: institutions should plan SIS/CRM integrations and campaign ownership before vendor scale-up.","governanceImplications":"Centralized institutional ownership and cross-unit coordination are described as critical; tailored outreach requires data sharing and clear campaign governance. Interpretation: treat chatbot campaigns as enterprise student-communications governance, not a single-office pilot.","securityPrivacyImplications":"Uses university administrative records (enrollment, aid, demographics) to target messages; source does not detail FERPA controls. Interpretation: procurement and privacy review should define allowed data elements, retention, and student notice before scaling personalized outreach.","caveats":"Working paper (July 2026), not peer-reviewed journal version. Tool is non-generative AI texting—transfer to generative tutoring/advising agents is limited. Subgroup nulls reported as available upon request in text. COVID-era conditions and CSUN context may limit external validity. Spillover cannot be fully ruled out though authors argue design reduces contamination risk.","streamIds":["student-success"],"roles":{"sales":"Sales framing: do not sell AI student-success chatbots as retention or GPA engines on this evidence. CSUN’s four-year RCT shows receptivity and better completion of acute administrative tasks, with precisely estimated nulls on persistence, credits, graduation, and GPA—including equity subgroups. Use the paper to set honest scopes for registrar/financial-aid workflow support and to challenge vendors that equate message response rates with graduation ROI. Bound any engagement to administrative-task KPIs and multi-year operating ownership, not guaranteed retention lift.","engineering":"Engineering framing: architecture must assume centralized campaign ownership plus flexible, data-triggered messaging into existing student-information workflows. The study’s value is operational—sustained SIS-linked outreach—not generative RAG advising. Prerequisites include reliable student contact data, unit-specific campaign APIs, and audit logging of who receives what. Constraints: non-generative scope; jurisdictional FERPA limits on demographic targeting. Proof of value should replicate task-completion lifts while pre-registering null-tolerant tests for persistence and GPA.","delivery":"Delivery framing: delivery work is institutional operating-model change—Registrar/student-success ownership, campaign calendars, and cross-office content—not a one-term chatbot install. Dependencies include data-sharing agreements and staffing for message design during shocks (e.g., pandemic). Skills adoption centers on communications analytics and advisor escalation paths. Acceptance criteria should privilege time-sensitive task completion with explicit non-claims on persistence; risks include over-promising retention and equity harm if targeting lacks consent and review."},"roleDetails":{"sales":{"customerProblem":"Public universities need scalable outreach for enrollment holds, aid forms, and registration without evidence that chatbots alone raise persistence or GPA.","stakeholders":["Student success / retention leaders","Registrar and financial aid","CIO / student communications","Institutional research"],"discoveryQuestions":["Which acute administrative tasks currently miss deadlines at scale?","Who owns multi-year campaign governance and data access?","What persistence/GPA claims has any vendor already made?","How will equity-subgroup outcomes be monitored?"],"valueHypothesis":"Bounded value may exist for time-sensitive administrative completion and sustained student receptivity under centralized ownership; this source does not support retention or GPA guarantees.","potentialEngagement":"Interpretation: a discovery-and-metrics workshop to map administrative campaigns to measurable task KPIs and to design a null-tolerant evaluation plan—not a retention-ROI sale.","unsupportedClaims":["Guaranteed retention or graduation lift","GPA improvement from messaging alone","Equity-closing effects for Pell/first-gen students","Transfer of these nulls-as-success to generative tutoring products"]},"engineering":{"fit":"Fit for non-generative, centrally orchestrated SMS/chat outreach tied to SIS events; poor fit as a standalone generative advising agent.","architectureIntegration":"Integrate campaign engine with registrar/financial-aid flags, consent/opt-out, and logging; keep human escalation for complex cases.","prerequisites":["Authoritative student contact channels","Shared administrative data definitions","Central campaign owner","Opt-out and audit capabilities"],"constraints":["U.S. public-university FERPA/data-minimization rules","Non-generative tool scope in the study","CSUN-specific operating context","Potential message fatigue over multi-year exposure"],"security":"Minimize demographic fields used for targeting; encrypt transit/storage of SIS extracts; document retention for message logs.","proofOfValue":["Pre-registered task-completion endpoints","Balanced RCT or stepped-wedge where feasible","Persistence/GPA monitored as secondary with MDES stated","Subgroup reporting for Pell/first-gen/Hispanic/female"]},"delivery":{"work":"Stand up centralized campaign operations, unit playbooks, and multi-year content refresh rather than a point chatbot deployment.","dependencies":["Registrar/financial-aid data feeds","Student communications staffing","IR evaluation capacity","Vendor or internal messaging platform"],"ownership":"Interpretation: name a durable institutional owner (e.g., Registrar or student-success ops), not a temporary grant PI.","skillsAdoption":"Train staff on campaign design, escalation, and interpreting task metrics without over-claiming academic impact.","governanceCheckpoints":["Data-sharing approval","Message content review","Annual outcome evaluation including nulls","Equity-subgroup readout"],"acceptanceCriteria":["Documented lift on specified administrative tasks","No unverified retention claim in go-live materials","Published evaluation plan with power assumptions","Opt-out and complaint path live"],"risks":["Vendor marketing converts task metrics into graduation ROI","Privacy complaints from targeted outreach","Campaign ownership fragments after pilot funding ends","Message volume harms trust"]}},"retrievedAt":"2026-09-15T14:15:11Z","enrichedAt":"2026-09-15T14:17:30Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Large share of analytic sample was Pell-eligible, Hispanic, and first-generation; nulls on persistence/GPA for those subgroups caution against claiming equity gains from outreach alone. Staff workload shifts toward campaign coordination across offices.","procurementImplications":"Procure against administrative-task SLAs and multi-year operating costs for centralized ownership; do not accept retention/GPA guarantees. Require evaluation rights and data-export for IR.","operatingModelImplications":"Success hinges on centralized ownership, flexible communication, and cross-unit coordination—an operating-model change more than a tool install.","sourceVerification":{"openedUrl":"https://edworkingpapers.com/sites/default/files/ai26-1409_v2.pdf","referenceExcerpt":"impacts are concentrated in improved completion of time-sensitive administrative tasks, with no detectable effects on academic performance or persistence.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}