{"resourceId":"susnjak-ai-guided-support-doubly-robust-arxiv","versions":[{"version":"external-ab9281bcc03778714c4073ac52ab4eb0517ef6bdb207c83e3156e785e215da83","resource":{"id":"susnjak-ai-guided-support-doubly-robust-arxiv","title":"Doubly robust evaluation: AI-guided proactive support cuts failures and lifts grades; completion speed weaker","organization":"Massey University (Auckland) / arXiv preprint","sector":"Higher education student support / learning analytics","geography":"New Zealand large university (Massey University)","publishedAt":"arXiv preprint 11 December 2025 (v1)","publicationDate":"2025-12-11","eventDate":null,"sourceName":"arXiv cs.CY","sourceLabel":"Susnjak, Bakhshov & Mathrani — Stabilising Learner Trajectories (AI-guided support)","sourceUrl":"https://arxiv.org/pdf/2512.11154","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["operating-model","data-security","governance-procurement","knowledge-work"],"finding":"Observational causal evaluation of an institutional AI learning-analytics intervention that produces daily probability-of-success scores used by a central student-support team for proactive outreach to at-risk undergraduates. Using blocked/calipered propensity-score matching and doubly robust estimation with bootstrap uncertainty, authors report the intervention reduced course failure rates and raised cumulative grades (treated students estimated ~8.8 grade points higher on average than matched controls; treated students failed about 0.78 fewer courses on average). Effects on speed of qualification completion were positive but statistically constrained. Activity Theory framing: AI risk scores mediate support work and stabilize trajectories, but institutional secondary contradictions limit translation into faster completion. Ethics approval Massey HE014–OM3 23/42.","sledRelevance":"Newly relevant after 2026-09-16 ACC Digital Twin reporting (no outcomes): this unarchived evaluation supplies Pass A measured effects for AI-guided proactive support—the advising/early-alert gap called out in prior editions—while Pass B notes non-RCT design and weak completion-speed effects.","evidence":"Source findings: PSM + doubly robust ATT-style estimates; highlights claim reduced failures and improved grades/pass rates; AI model performance metrics reported (e.g., F1/AUC on risk detection). Limitations: not a randomized trial; residual confounding possible despite sensitivity analysis; single NZ university; preprint; completion-speed effects constrained; immortal-time addressed via dynamic scores but design remains observational.","architectureImplications":"Interpretation: couple daily risk scoring with human outreach workflows and auditable decision logs; measure failures/grades and progression separately. Integration: SIS/LMS features → risk service → support CRM caseing.","governanceImplications":"Interpretation: govern threshold rules, appeal/override, and when scores may drive mandatory contact; Activity Theory points to rule/workflow redesign—not model accuracy alone.","securityPrivacyImplications":"Risk scores and outreach logs are sensitive education records. Interpretation: least-privilege for support staff, retention limits, student notice, and jurisdictional privacy controls (NZ Privacy Act analogue to FERPA reviews for U.S. transfer).","caveats":"December 2025 preprint (older than last run) included because AI-guided support outcome evidence was a stated archive gap and URL was check_sources-unknown. Observational identification ≠ RCT. Grade-point metric may be local scale. Transfer to U.S. community colleges requires re-estimation.","streamIds":["student-success"],"roles":{"sales":"Sales framing: for early-alert/AI advising, sell measured stabilization (fewer failures, higher grades) with explicit limits on completion-speed claims. This Massey doubly robust study estimates ~8.8-point higher mean grades and ~0.78 fewer failed courses for AI-guided support recipients versus matched peers—but is observational, single-institution, and weak on faster qualification completion. Unsupported: guaranteed graduation-rate jumps or fully automated advising.","engineering":"Engineering framing: architecture is risk-score service + support CRM + human judgment, with PSM/doubly robust evaluation hooks. Prerequisites: rich pre-intervention covariates, daily score freshness, and case management. Constraints: confounding; NZ context. Proof of value: replicate failure/grade ATT with sensitivity to unobserved confounding and separate progression KPIs.","delivery":"Delivery framing: delivery work is outreach operations redesign—who contacts whom under which score thresholds—not only ML ops. Ownership: student support leadership with data science. Risks: alert fatigue, discriminatory thresholding, and over-automating without Activity Theory–aligned rule changes."},"roleDetails":{"sales":{"customerProblem":"Early-alert AI is procured without causal-ish evidence on failures/grades versus progression.","stakeholders":["VP student success","Advising leadership","IR/analytics","Privacy/compliance"],"discoveryQuestions":["Is evaluation randomized or matched observational?","Which KPIs—failures, GPA, completion speed?","Who may override risk scores?","What student notice exists for scoring?"],"valueHypothesis":"AI-guided proactive outreach can reduce failures and raise grades for high-risk students when humans act on fresh scores, without assuming faster degree completion.","potentialEngagement":"Matched-cohort evaluation design for an existing early-alert program, plus threshold/workflow redesign workshop.","unsupportedClaims":["RCT-level causal certainty from this study alone","Faster graduation guaranteed","Advisor headcount elimination"]},"engineering":{"fit":"Fits institutions with SIS/LMS data and a central support team using predictive scores for outreach.","architectureIntegration":"Daily PoS model → secure score store → support CRM queues → outcome warehouse for doubly robust evaluation.","prerequisites":["Historical covariates","Intervention timestamping","CRM adoption","Ethics/privacy approval"],"constraints":["Observational bias","Local grading scales","Staff capacity to act on alerts"],"security":"Risk scores are high-sensitivity; encrypt, RBAC, and audit every access/override.","proofOfValue":["Pre-specify ATT estimands","Sensitivity to unobserved confounding","Separate completion-speed analysis"]},"delivery":{"work":"Redesign outreach protocols, train advisors on score interpretation, and stand up evaluation pipelines.","dependencies":["Advisor staffing","Data engineering","Ethics board","Student communications"],"ownership":"Director of student support with analytics lead.","skillsAdoption":"Advisors need score literacy and bias awareness; analysts need causal-evaluation skills.","governanceCheckpoints":["Threshold policy approval","Privacy DPIA/FERPA review","Equity audit of who is flagged"],"acceptanceCriteria":["Documented outreach SLAs by risk band","Failure/grade dashboards live","Override audit trail enabled"],"risks":["Alert fatigue","Stigmatizing outreach","Model drift without monitoring"]}},"retrievedAt":"2026-09-18T03:06:28Z","enrichedAt":"2026-09-18T03:15:52Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Interpretation: proactive outreach may help students who do not self-refer, including some disabled or first-in-family students, but risk models can encode bias—require equity audits and accessible contact channels.","procurementImplications":"Interpretation: procure early-alert as score+workflow+evaluation package; contract for exportable covariates/outcomes to enable doubly robust monitoring.","operatingModelImplications":"Interpretation: shift support from reactive referral to score-triggered proactive work with clear division of labour; revise rules when completion-speed effects stall.","sourceVerification":{"responseId":"curl-arxiv-2512-11154-2026-09-17","openedUrl":"https://arxiv.org/pdf/2512.11154","referenceExcerpt":"treated students will attain mean grades that are, on average, 8.8 points higher","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}