{"resourceId":"berk-arxiv-probation-risk-conformal-202609","versions":[{"version":"external-4242d602cb0c264ad8bc980d5ad9ce3b9af66a9c186d3111c172e496b36da046","resource":{"id":"berk-arxiv-probation-risk-conformal-202609","title":"Berk (arXiv): Mondrian conformal sets curb coin-flip probation risk class assignments","organization":"University of Pennsylvania Department of Criminology (Richard Berk)","sector":"Corrections / probation risk assessment","geography":"United States — large anonymized American city probation/parole agency","publishedAt":"September 2026 (arXiv:2609.22718)","publicationDate":null,"eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint with full methods, tables, and probation case study","sourceUrl":"https://arxiv.org/abs/2609.22718","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","governance-procurement","operating-model","data-security"],"finding":"Berk's September 2026 arXiv paper analyzes a random-forest probation/parole re-arrest risk tool (2019–2024 data; high/moderate/low classes) for a large U.S. city agency. Standard argmax classification often assigns labels when top class scores are nearly tied—'essentially being assigned by a coin flip.' Using Mondrian conformal prediction sets (α=0.25; target coverage 0.75), singleton High-risk forecasts were empirically about 45% accurate (improved vs ~37% confusion-table prediction success for Predicted High), Low-risk singletons ~88–89% accurate, and overall empirical coverage about 0.75–0.77. The paper recommends deciding from conformal sets—not raw classifier argmax—and treating multi-label sets as explicit uncertainty rather than forced single labels.","sledRelevance":"Why-now academic methods paper for SLED probation/parole AI: same-week as HMCTS admin-AI pilot and prior-night custody/probation AI governance coverage; shows measurable unreliability of high-risk labels without uncertainty sets.","evidence":"Opened arXiv abs + HTML full text; confirmed abstract coin-flip language, data years, stratified RF, α=0.25, Table 5 High coverage ~0.45, Low ~0.88, overall ~0.77, and policy recommendation to use conformal sets for decisions.","architectureImplications":"Risk services should emit prediction sets and conformity thresholds, not only argmax labels; UI must display multi-class uncertainty to officers.","governanceImplications":"Agency policy should forbid acting on near-tied argmax scores without conformal/uncertainty rules; document coverage α as a stakeholder choice.","securityPrivacyImplications":"Probation datasets are sensitive; paper notes public-but-hard-to-obtain data and excluded race/zip/juvenile priors for political constraints.","caveats":"Preprint; anonymized city; outcome taxonomy and excluded predictors are agency-specific; not a vendor product evaluation.","streamIds":["public-safety"],"roles":{"sales":"Sales should open on unreliable high-risk labels—not model marketing accuracy. Berk's probation study shows random-forest argmax can assign high/moderate/low when scores are nearly tied (coin-flip territory), while Mondrian conformal sets at 75% target coverage produce singleton High forecasts only ~45% accurate—and Low singletons near 90%. Position probation AI work as a bounded uncertainty-governance engagement: prediction-set decision rules, α selection with stakeholders, and step-up/step-down procedures when sets are multi-label. Do not claim conformal methods guarantee fairness or public-safety gains.","engineering":"Engineering fit is wrapping existing probability-output risk classifiers with Mondrian conformal calibration (train/calibrate/forecast splits) and serving set-valued outputs to case-management UIs. Prerequisites: held-out calibration data, stakeholder-chosen α, and exchangeability checks over time. Constraints: class imbalance, political predictor exclusions, and costly over-prediction of high/moderate supervision. Proof of value: empirical coverage near target α and reduction in forced single labels when scores are near-tied.","delivery":"Delivery work is risk-tool modernization: implement conformal serving, retrain officers on set-valued outputs, rewrite supervision assignment SOPs for multi-label sets, and schedule reassessment. Ownership sits with probation leadership for policy and analytics for model ops. Acceptance criteria should include documented α, UI showing set contents, and audited decisions that do not silently argmax near-ties."},"roleDetails":{"sales":{"customerProblem":"Probation agencies ship risk labels that look decisive when classifier scores are nearly tied, creating unfair and error-prone supervision assignments.","stakeholders":["Chief probation officer","Agency analytics","County counsel","Supervision managers","Community advisory","IT/CMS"],"discoveryQuestions":["Do you act on argmax when top two class probabilities differ by <5%?","Is coverage/uncertainty a stakeholder-chosen parameter?","How are multi-label forecasts handled in SOPs?","What is measured accuracy for High-risk labels in holdout?"],"valueHypothesis":"Bounded value is uncertainty-aware decision rules around existing classifiers—not a new black-box risk score product.","potentialEngagement":"8–12 week conformal-prediction pilot on an existing probation risk model with policy rewrite.","unsupportedClaims":["Guaranteed recidivism reduction","That conformal sets eliminate racial disparity","Vendor accuracy claims without local calibration"]},"engineering":{"fit":"Post-hoc Mondrian conformal layer on probability classifiers; set-valued API to CMS.","architectureIntegration":"Train RF → calibration conformity thresholds by class → forecast sets → CMS displays set + α → officer decision log.","prerequisites":["Split calibration data","Chosen α","Exchangeability monitoring","CMS UI fields"],"constraints":["Imbalanced outcomes","Excluded sensitive predictors","Supervision cost of over-predicting High/Moderate"],"security":"Protect offender features and scores; limit vendor access to identifiable probation records.","proofOfValue":["Empirical coverage ≈ 1−α","Near-tied cases emit multi-label sets","Decision logs reference set contents"]},"delivery":{"work":"Calibrate conformal layer, update SOPs for multi-label sets, train officers, and monitor coverage drift.","dependencies":["Analytics capacity","CMS change","Policy approval","Training"],"ownership":"Probation leadership owns supervision policy; analytics owns model ops; officers own case-level decisions.","skillsAdoption":"Train that 'High' singleton is selective and that {High, Moderate} means uncertainty—not a forced High.","governanceCheckpoints":["α selection workshop","Pilot coverage review","Annual reassessment"],"acceptanceCriteria":["Set-valued UI live","SOP for multi-label sets","Published local accuracy note"],"risks":["Over-supervision costs","Misread multi-label sets","Distribution shift breaking exchangeability"]}},"retrievedAt":"2026-10-05T02:59:21Z","enrichedAt":"2026-10-05T02:59:41Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Officers and clients need plain explanations of prediction sets and why some cases get multi-label uncertainty instead of a single risk stamp.","procurementImplications":"Require vendors to support calibration exports and set-valued outputs; do not procure argmax-only risk SKUs without uncertainty tooling.","operatingModelImplications":"Supervision intensity decisions should reference conformal sets and documented tie-break policies, not silent argmax.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.22718","referenceExcerpt":"when they are much the same, the outcome class is essentially being assigned by a coin flip","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}