{"resourceId":"gsu-course-chatbot-outreach-rct-june-2026","versions":[{"version":"external-ac81a7103efea5a6d2ed8987f7e8d4be3ae8e096075c8f7da39cc5445cd15842","resource":{"id":"gsu-course-chatbot-outreach-rct-june-2026","title":"Course chatbot trial finds bounded grade gains with weaker adjusted evidence","organization":"Katharine Meyer and colleagues; Brookings, Brown, Georgia State and collaborating universities","sector":"Higher education teaching and student support","geography":"Georgia, United States","publishedAt":"June 2026 manuscript version; exact day unknown","publicationDate":null,"eventDate":null,"sourceName":"Let’s Chat: Leveraging Chatbot Outreach for Improved Course Performance","sourceLabel":"Original academic working paper, June 2026 version","sourceUrl":"https://edworkingpapers.com/sites/default/files/ai22_564_v3.pdf","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","data-security","governance-procurement","accessibility-workforce","operating-model","infrastructure"],"finding":"Non-generative course outreach improved the A/B grade threshold, but evidence weakens after multiple-comparison correction and does not establish transferable learning.","sledRelevance":"Interpretation: Useful U.S. public-university evidence for testing course navigation support within an established student-service workflow.","evidence":"Blocked randomized intent-to-treat study: 1,568 Government and 915 Microeconomics students, 2021–23. Table 2: pooled A/B control mean 0.61; adjusted-covariate effect +0.04, SE 0.018; multiple-comparison-adjusted p=.090. Baseline included usual email and university retention messaging. The intervention combined targeted texts, a curated response bank and TA oversight.","architectureImplications":"Interpretation: Map consent, roster updates, assignment status, message delivery and escalation before choosing a platform. A modern LLM replacement would require fresh validation; this intervention does not validate autonomous agents or select cloud, local or hybrid hosting.","governanceImplications":"Interpretation: Register endpoints and multiple-testing rules before rollout, and prevent marketing from elevating a favorable threshold over other reported outcomes.","securityPrivacyImplications":"Interpretation: Minimize grade details in texts, verify recipient identity, restrict staff access and test opt-out propagation. Treat unexpected sensitive disclosures as a human escalation need.","caveats":"One institution and texting opt-ins; pooled numeric-grade effect nonsignificant. No strong spillover or subsequent-term effect; no formal cost-effectiveness analysis. The text's significance description needs qualification against Table 2. Grades are not a direct durable-learning test.","streamIds":["student-success"],"roles":{"sales":"Interpretation — A course leader may struggle to reach students before small missed tasks accumulate. Include faculty, advising, student representatives, institutional research and IT in discovery. Ask which tasks are missed, what outreach already exists and who answers replies. Offer a bounded pilot in one stable course with an agreed comparator and complete cost capture. The value hypothesis is more timely access to existing support. This trial justifies testing that hypothesis locally, not guaranteeing an A/B increase. Avoid promising retention gains, staff reductions or a return on investment. A campus without reliable student data and response capacity needs readiness work first.","engineering":"Interpretation — Design a read-only connection to course status and an approved response queue before enabling outbound targeting. Prerequisites include accurate rosters, consent records, faculty-approved deadlines and a maintained contact directory. Test stale enrollment, incorrect recipient mapping, ambiguous questions and withdrawals using synthetic records. Compare the full workflow with current outreach, including unresolved questions and staff review time. Proposed proof of value: report prespecified course outcomes and uncertainty alongside routing accuracy and response delays. A generative copilot may help staff draft content, but it needs separate factual review and must not silently alter grades, enrollment or financial records.","delivery":"Interpretation — Assign an academic sponsor, a service lead and an independent evaluator. Build message templates, an escalation rota, correction procedures and student orientation before enrollment. Dependencies include approved data handling, accessible communication and staff training in sensitive disclosures. Proposed acceptance criteria: every sampled recipient mapping and opt-out test passes, every message has an accountable content owner, and all prespecified outcomes and operating costs are reported before expansion. These are proposed local gates, not study results. Watch for messaging fatigue, delayed escalation and selective participation. Keep alternative contact routes available and review whether support demand exceeds staffing."},"retrievedAt":"2026-09-13T03:01:24Z","enrichedAt":"2026-09-13T03:01:49Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Interpretation: Offer accessible alternative channels and include student-support labor in capacity planning; message delivery alone does not demonstrate equitable reach.","procurementImplications":"Interpretation: Price implementation, integration, message volume, monitoring and exit costs; require auditable routing and exportable records.","operatingModelImplications":"Interpretation: Keep course faculty accountable for message content and a staffed service accountable for questions and correction.","updateExplanation":"Not found in all 247 archive resources or targeted title search. Explicit June-version evidence backfill adds a controlled U.S. course-support study and its adjusted-statistics caveat, not a new September event.","sourceVerification":{"openedUrl":"https://edworkingpapers.com/sites/default/files/ai22_564_v3.pdf","referenceExcerpt":"We did not conduct a formal cost-effectiveness analysis for this study.","promptVersion":"sled-research-v3.1","model":null,"basis":"agent-reported inspection"}}}]}