From the Public Safety edition of October 20, 2026
Vendor-run study: an AI workflow matched an expert-adjudicated reference on 90.21% of rubric decisions when grading community corrections supervision visits, but on only ten development visits
RePath Labs (study authors) with expert graders; transcripts from multiple community corrections agencies · Corrections / community supervision (probation and parole) · United States (agencies not named)
- Publisher
- OSF Preprints
- Original publication
- Posted on OSF Preprints October 15, 2026 (white paper version 1.0 dated September 30, 2026)
- Source retrieved
- 2026-10-21
What happened
Three expert graders adjudicated 43 rubric items across ten supervision-visit transcripts (five adult, five juvenile), reaching unanimous decisions on 429 of 430 items. Against that reference, the experts’ initial independent ratings agreed on 49.65%–86.01% of decisions, while the selected GPT-6 Astra workflow (three runs, majority vote) agreed on 90.21% (387/429), kappa 0.832, mean absolute visit-score error 4.57 points. Four other workflows reached 80.89%–85.55%. All ten visits were used for development, so accuracy on new visits is unmeasured. The authors limit intended use to coaching, not risk prediction, liberty decisions, sanctions or discipline.
Why it matters
Corrections agencies can audit only a small share of supervision visits for core correctional practice. This is a concrete, measured attempt to use AI to review recorded visits against a shared standard, with unusually explicit limits on use. It is vendor-run and small, so it supports pilot design, not procurement decisions.
Evidence and measured results
Opened the full white paper PDF from OSF (download link) and read the executive summary, scope, limitations, responsible-use recommendations, data and competing-interest sections. OSF API metadata shows publication on Oct. 15, 2026. RePath Labs designed, funded and conducted the study and has a commercial interest; the human comparison uses experts’ initial ratings, while the AI workflow was developed against the reference; transcripts are not public.
Limitations and uncertainty
Vendor-funded preprint, not peer reviewed; ten visits, all used in development; reference is expert consensus, not ground truth; model versions change; no outcome data on officer practice or recidivism.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-21; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Community corrections agencies audit only a small fraction of supervision visits, and expert reviewers disagree.
- Stakeholders
- Chief probation officer
- Training and quality assurance manager
- Supervisors
- Officers’ union
- Agency counsel
- Discovery questions
- Are supervision visits recorded and transcribed?
- Which practice model and rubric does the agency use?
- How many visits are reviewed today?
- What uses of scores would be prohibited?
- Value hypothesis
- Validated AI grading could expand coaching review coverage if it meets agreement thresholds on the agency’s own visits.
- Potential engagement
- Validation pilot on held-out local visits against an agency-adjudicated reference, coaching use only.
- Unsupported claims
- Reduced recidivism
- Accuracy on unseen visits
- Suitability for discipline or liberty decisions
Pre-sales engineering
Role takeaway
- Fit
- Agencies with recorded supervision visits and a defined core-correctional-practice rubric.
- Architecture and integration
- Transcription feeds item-level grading with evidence citations into a supervisor coaching dashboard.
- Prerequisites
- Visit recordings and transcription
- De-identification process
- Local adjudicated reference set
- Constraints
- Model versions change
- Transcripts miss tone and nonverbal conduct
- State privacy rules on recordings
- Security
- De-identify transcripts, restrict access to supervisors, and record which model configuration produced each grade.
- Proof of value
- Proposed: measure agreement on visits not used in development
- Proposed: sample unanimous-run items for hidden errors
Delivery
Role takeaway
- Work
- Build a local reference, validate grading out of sample, and integrate grades into supervisor coaching.
- Dependencies
- Expert reviewer time
- Policy on permitted uses
- Recording infrastructure
- Ownership
- Community supervision training and quality assurance unit
- Skills and adoption
- Supervisors trained to use cited evidence in coaching and to accept officer challenges to marks.
- Governance checkpoints
- Written prohibition on disciplinary use
- Re-validation on any model or rubric change
- Periodic sampled human review
- Acceptance criteria
- Proposed: at least 85% agreement with local reference on held-out visits
- Proposed: every grade links to transcript evidence
- Risks
- Scores repurposed for discipline
- Performance drops on new visits
- Vendor lock-in to a model configuration
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Transcript-based grading using a commercial model with three-run majority vote, item-level citations to transcript evidence, and model-specific validation.
Governance
Who approves, reviews and stays accountable for outcomes?
Authors recommend visible evidence, the ability to contest marks, monitoring items where runs agree, and re-validation whenever the model or policy changes.
Security and privacy
What data, permissions and controls need testing?
Transcripts of real supervision visits were de-identified before grading; recordings of probation meetings contain sensitive justice-system information.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Intended to support officer coaching; risk that scores are misused for discipline, which the authors exclude from validated use.
Procurement
What should contracts, pricing and exit terms secure?
Buyers should require out-of-sample validation on their own visits and contract terms prohibiting use for discipline or liberty decisions without separate validation.
Operating model
Which teams own the service once it runs?
Shifts supervision quality review from small manual audits toward broader automated review with human coaches and sampled checks.
Publication history
- 2026-10-20Public Safety · Issue 426 resources
Stable resource ID: repath-ai-grading-community-corrections-supervision-visits-20261015