From the Public Safety edition of October 2, 2026
Berk (arXiv): Mondrian conformal sets curb coin-flip probation risk class assignments
University of Pennsylvania Department of Criminology (Richard Berk) · Corrections / probation risk assessment · United States — large anonymized American city probation/parole agency
- Publisher
- arXiv
- Original publication
- September 2026 (arXiv:2609.22718)
- Source retrieved
- 2026-10-05
What happened
Berk's September 2026 arXiv paper analyzes a random-forest probation/parole re-arrest risk tool (2019–2024 data; high/moderate/low classes) for a large U.S. city agency. Standard argmax classification often assigns labels when top class scores are nearly tied—'essentially being assigned by a coin flip.' Using Mondrian conformal prediction sets (α=0.25; target coverage 0.75), singleton High-risk forecasts were empirically about 45% accurate (improved vs ~37% confusion-table prediction success for Predicted High), Low-risk singletons ~88–89% accurate, and overall empirical coverage about 0.75–0.77. The paper recommends deciding from conformal sets—not raw classifier argmax—and treating multi-label sets as explicit uncertainty rather than forced single labels.
Why it matters
Why-now academic methods paper for SLED probation/parole AI: same-week as HMCTS admin-AI pilot and prior-night custody/probation AI governance coverage; shows measurable unreliability of high-risk labels without uncertainty sets.
Evidence and measured results
Opened arXiv abs + HTML full text; confirmed abstract coin-flip language, data years, stratified RF, α=0.25, Table 5 High coverage ~0.45, Low ~0.88, overall ~0.77, and policy recommendation to use conformal sets for decisions.
Limitations and uncertainty
Preprint; anonymized city; outcome taxonomy and excluded predictors are agency-specific; not a vendor product evaluation.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-05; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Probation agencies ship risk labels that look decisive when classifier scores are nearly tied, creating unfair and error-prone supervision assignments.
- Stakeholders
- Chief probation officer
- Agency analytics
- County counsel
- Supervision managers
- Community advisory
- IT/CMS
- Discovery questions
- Do you act on argmax when top two class probabilities differ by <5%?
- Is coverage/uncertainty a stakeholder-chosen parameter?
- How are multi-label forecasts handled in SOPs?
- What is measured accuracy for High-risk labels in holdout?
- Value hypothesis
- Bounded value is uncertainty-aware decision rules around existing classifiers—not a new black-box risk score product.
- Potential engagement
- 8–12 week conformal-prediction pilot on an existing probation risk model with policy rewrite.
- Unsupported claims
- Guaranteed recidivism reduction
- That conformal sets eliminate racial disparity
- Vendor accuracy claims without local calibration
Pre-sales engineering
Role takeaway
- Fit
- Post-hoc Mondrian conformal layer on probability classifiers; set-valued API to CMS.
- Architecture and integration
- Train RF → calibration conformity thresholds by class → forecast sets → CMS displays set + α → officer decision log.
- Prerequisites
- Split calibration data
- Chosen α
- Exchangeability monitoring
- CMS UI fields
- Constraints
- Imbalanced outcomes
- Excluded sensitive predictors
- Supervision cost of over-predicting High/Moderate
- Security
- Protect offender features and scores; limit vendor access to identifiable probation records.
- Proof of value
- Empirical coverage ≈ 1−α
- Near-tied cases emit multi-label sets
- Decision logs reference set contents
Delivery
Role takeaway
- Work
- Calibrate conformal layer, update SOPs for multi-label sets, train officers, and monitor coverage drift.
- Dependencies
- Analytics capacity
- CMS change
- Policy approval
- Training
- Ownership
- Probation leadership owns supervision policy; analytics owns model ops; officers own case-level decisions.
- Skills and adoption
- Train that 'High' singleton is selective and that {High, Moderate} means uncertainty—not a forced High.
- Governance checkpoints
- α selection workshop
- Pilot coverage review
- Annual reassessment
- Acceptance criteria
- Set-valued UI live
- SOP for multi-label sets
- Published local accuracy note
- Risks
- Over-supervision costs
- Misread multi-label sets
- Distribution shift breaking exchangeability
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Risk services should emit prediction sets and conformity thresholds, not only argmax labels; UI must display multi-class uncertainty to officers.
Governance
Who approves, reviews and stays accountable for outcomes?
Agency policy should forbid acting on near-tied argmax scores without conformal/uncertainty rules; document coverage α as a stakeholder choice.
Security and privacy
What data, permissions and controls need testing?
Probation datasets are sensitive; paper notes public-but-hard-to-obtain data and excluded race/zip/juvenile priors for political constraints.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Officers and clients need plain explanations of prediction sets and why some cases get multi-label uncertainty instead of a single risk stamp.
Procurement
What should contracts, pricing and exit terms secure?
Require vendors to support calibration exports and set-valued outputs; do not procure argmax-only risk SKUs without uncertainty tooling.
Operating model
Which teams own the service once it runs?
Supervision intensity decisions should reference conformal sets and documented tie-break policies, not silent argmax.
Publication history
- 2026-10-02Public Safety · Issue 284 resources
Stable resource ID: berk-arxiv-probation-risk-conformal-202609