{"resourceId":"wang-risk-adaptive-genai-feedback-20260924","versions":[{"version":"external-38bf826a85e713eb5754b1f4bfd9c3c7f87e3c3e890fca3282a538f0604e64bb","resource":{"id":"wang-risk-adaptive-genai-feedback-20260924","title":"UCL/ProgFeed study: risk-adaptive, evidence-constrained GenAI feedback for intro programming","organization":"UCL Institute of Education (Shihao Wang); ProgFeed / UMass ML4Ed dataset","sector":"Higher education computer science / learning analytics","geography":"United Kingdom analysis of U.S. Fall 2025 intro programming course traces (215 students)","publishedAt":"arXiv 24 September 2026; PDF marked Version September 25, 2026 submitted to Education Sciences","publicationDate":"2026-09-24","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint submitted to Educ. Sci.","sourceUrl":"https://arxiv.org/pdf/2609.29874","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["developers-agents","data-security","governance-procurement","operating-model","knowledge-work"],"finding":"Using 2,993 failed-submission states from 215 intro-programming students (student-disjoint splits), a validation-selected logistic model predicted next-two-attempt persistent failure at test PR-AUC 0.550 and ROC-AUC 0.681. Four matched GenAI feedback conditions (F1–F4) were generated for 136 cases; after repair and evidence gating, 519 of 544 messages (95.4%) contained required components while unverifiable code quotes were stripped. A fixed-threshold sequential policy using calibrated risk selected 17.8% of eligible test states and captured 25.2% of observed persistent failures. The paper proposes evidence-gated progressive assistance—from self-checks to localized hints—keeping prediction, decision, generation, and learning outcomes as distinct evaluation claims. It does not yet report a prospective RCT of learning gains from the new F1–F4 messages.","sledRelevance":"Directly useful to CS and STEM student-success teams deploying GenAI feedback or early-alert copilots: shows how to couple calibrated risk with source-bounded progressive hints under instructor-capacity limits, rather than blasting every failure with full solutions. Complements tutoring-equivalence news by specifying operating controls against over-assistance.","evidence":"Retrospective computational study on public ProgFeed dataset (CC BY 4.0); student-disjoint 60/20/20; multiple model families; paired within-case generation with Qwen2.5-Coder-1.5B-Instruct; structural/source gates; fixed-capacity and sequential policy replay. Secondary outcomes include unmodified retry and related-task performance windows.","architectureImplications":"Separate services for (1) time-valid risk scoring, (2) capacity-aware trigger policy, (3) evidence-bounded message generation with deterministic source verification, and (4) human-review queue for unresolved structure. Prefer interpretable baselines (logistic) as auditable defaults alongside deeper models for history-sensitive behaviors.","governanceImplications":"Require documented intervention budgets, threshold freezes before test use, and distinct claims for prediction vs policy vs message quality vs learning. Progressive assistance should be policy-approved so high-specificity hints are reserved for sustained failure—not default answer dumps.","securityPrivacyImplications":"Source data are de-identified research consents; campus production systems must map failed submissions and code to FERPA/school-official controls, minimize retention of full code in LLM prompts, and block training on student code without contract. Source-gating reduces hallucinated line references but does not eliminate privacy risk in prompts.","caveats":"Moderate discrimination (PR-AUC ~0.55); no prospective learner-outcome trial of F1–F4; single course/language; near-term related-task prediction weak; historical natural-language feedback association is observational within a randomized subset, not a causal proof of the new generator.","streamIds":["student-success"],"roles":{"sales":"For CS deans and student-success leaders drowning in failed autograder attempts, this framework sells an operating model—not a magic detector: calibrated risk picks whom to help, evidence gates what the bot may say, and assistance escalates from self-checks to localized hints under capacity caps. Bound claims to the ProgFeed retrospective metrics (PR-AUC 0.550; 17.8% alerts capturing 25.2% persistent failures; 95.4% structurally complete messages). Do not promise grade or retention lifts until a prospective pilot measures them.","engineering":"Engineering should implement student-disjoint trained risk models, validation-fitted calibration, frozen thresholds, and a generator that only quotes code present in the bounded record. Wire LMS or autograder webhooks to a decision service with explicit alert budgets and a human-review queue. Constraints include moderate predictive power and course-specific failure patterns dominating features. Proof-of-value is precision and capture under a stated budget plus human ratings of grounding—not chatbot satisfaction alone.","delivery":"Delivery should start with instructor-in-the-loop review of high-risk alerts, publish progressive-assistance rules to students, and add delayed unaided tasks the paper flags as future work. Ownership sits with CS course coordinators alongside learning-analytics and academic technology. Risks include alert fatigue, inequitable targeting when task identifiers dominate features, and mistaking structural message completeness for measured learning gains."},"roleDetails":{"sales":{"customerProblem":"Intro programming support cannot manually review every failed submission; unguarded GenAI risks answer dumping.","stakeholders":["CS department","teaching & learning","learning analytics","academic technology","student success"],"discoveryQuestions":["What is weekly failed-submission volume and instructor review capacity?","Are GenAI hints currently unconstrained?","Which outcomes define persistent failure locally?"],"valueHypothesis":"Capacity-aware, evidence-gated progressive feedback can concentrate limited support on persistent-failure risk without default full solutions.","potentialEngagement":"Retrospective shadow-mode risk policy on one CS1 course, then supervised progressive-hint pilot with human review.","unsupportedClaims":["Proven grade/retention lift from F1–F4 messages","PR-AUC suitable as a sole go-live gate","Cross-language/course portability without revalidation"]},"engineering":{"fit":"Autograder-backed intro programming courses with attempt histories.","architectureIntegration":"Webhook → feature store → calibrated risk → policy → bounded LLM generate → source gate → instructor queue.","prerequisites":["student-disjoint historical logs","problem/test metadata","prompt/context size limits"],"constraints":["moderate discrimination","task-heterogeneous failure rates","single-course evidence"],"security":"Prompt only bounded code/tests; prohibit training on student code; audit logs of risk, evidence pack, and message.","proofOfValue":["Precision/capture at agreed alert budget","% messages passing source gate","Instructor accept/edit rate"]},"delivery":{"work":"Shadow-mode scoring, threshold workshops with faculty, progressive-hint rubric, student notice, weekly alert triage.","dependencies":["autograder data quality","faculty time for review","privacy review"],"ownership":"Course coordinator with learning-analytics co-owner.","skillsAdoption":"Faculty training on escalation levels; TA triage playbooks; student literacy on self-checks vs hints.","governanceCheckpoints":["freeze thresholds before any auto-send","equity review of alert rates by subgroup","stop-ship if source-gate fail rate spikes"],"acceptanceCriteria":["Documented alert budget and capture metrics","Human alternative path for contested feedback","Plan for delayed unaided assessment"],"risks":["Over-alerting low-capacity staff","Students ignore self-checks","Feature leakage from future labels if rebuilds are sloppy"]}},"retrievedAt":"2026-09-27T03:23:30Z","enrichedAt":"2026-09-27T03:26:58Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Progressive assistance can reduce wait time for struggling programmers while preserving productive struggle; workforce shifts toward triage of high-risk states rather than answering every failure. Equity requires monitoring whether alerts concentrate on particular task types or student groups and offering non-AI help paths.","procurementImplications":"Buy platforms that expose calibrated risk, configurable budgets, evidence packs, and source verification—not only LLM chat wrappers. Contract for student-disjoint validation reports and prohibition on training on institution code without approval.","operatingModelImplications":"Run prediction→decision→generation as an auditable loop with weekly faculty review of thresholds and hint levels. Keep learning-outcome evaluation on a separate track from message fluency metrics.","sourceVerification":{"openedUrl":"https://arxiv.org/pdf/2609.29874","referenceExcerpt":"calibrated risk guides intervention timing, recorded evidence constrains feedback content","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}