{"resourceId":"repath-ai-grading-community-corrections-supervision-visits-20261015","versions":[{"version":"external-05dd78ed5bf710de46d84bd8022cf730fd54fd5db8957490aef0f10201530b1c","resource":{"id":"repath-ai-grading-community-corrections-supervision-visits-20261015","title":"Vendor-run study: an AI workflow matched an expert-adjudicated reference on 90.21% of rubric decisions when grading community corrections supervision visits, but on only ten development visits","organization":"RePath Labs (study authors) with expert graders; transcripts from multiple community corrections agencies","sector":"Corrections / community supervision (probation and parole)","geography":"United States (agencies not named)","publishedAt":"Posted on OSF Preprints October 15, 2026 (white paper version 1.0 dated September 30, 2026)","publicationDate":"2026-10-15","eventDate":null,"sourceName":"OSF Preprints","sourceLabel":"Establishing and Meeting a Reference Standard for AI Grading of Community Corrections Supervision Visits","sourceUrl":"https://osf.io/download/6abd613e284e0da501d0eb72/","evidenceClass":"vendor-claim","outcomeClass":"emerging","topics":["knowledge-work","governance-procurement","accessibility-workforce","data-security"],"finding":"Three expert graders adjudicated 43 rubric items across ten supervision-visit transcripts (five adult, five juvenile), reaching unanimous decisions on 429 of 430 items. Against that reference, the experts’ initial independent ratings agreed on 49.65%–86.01% of decisions, while the selected GPT-6 Astra workflow (three runs, majority vote) agreed on 90.21% (387/429), kappa 0.832, mean absolute visit-score error 4.57 points. Four other workflows reached 80.89%–85.55%. All ten visits were used for development, so accuracy on new visits is unmeasured. The authors limit intended use to coaching, not risk prediction, liberty decisions, sanctions or discipline.","sledRelevance":"Corrections agencies can audit only a small share of supervision visits for core correctional practice. This is a concrete, measured attempt to use AI to review recorded visits against a shared standard, with unusually explicit limits on use. It is vendor-run and small, so it supports pilot design, not procurement decisions.","evidence":"Opened the full white paper PDF from OSF (download link) and read the executive summary, scope, limitations, responsible-use recommendations, data and competing-interest sections. OSF API metadata shows publication on Oct. 15, 2026. RePath Labs designed, funded and conducted the study and has a commercial interest; the human comparison uses experts’ initial ratings, while the AI workflow was developed against the reference; transcripts are not public.","architectureImplications":"Transcript-based grading using a commercial model with three-run majority vote, item-level citations to transcript evidence, and model-specific validation.","governanceImplications":"Authors recommend visible evidence, the ability to contest marks, monitoring items where runs agree, and re-validation whenever the model or policy changes.","securityPrivacyImplications":"Transcripts of real supervision visits were de-identified before grading; recordings of probation meetings contain sensitive justice-system information.","caveats":"Vendor-funded preprint, not peer reviewed; ten visits, all used in development; reference is expert consensus, not ground truth; model versions change; no outcome data on officer practice or recidivism.","streamIds":["public-safety"],"roles":{"sales":"Sales framing: probation and parole leaders cannot review most supervision visits, and this study suggests AI could apply a shared rubric more consistently than unaided experts, on a small development set. The bounded engagement is an agency-specific validation pilot for coaching use only. The evidence cannot support claims of reduced recidivism, better officer performance, accuracy on new visits, or suitability for discipline.","engineering":"Engineering framing: the system grades transcripts item by item, cites evidence, and uses three runs with majority vote. Pre-sales should confirm recording and transcription quality, de-identification, where transcripts are processed, and whether model versions can be pinned. Proof must use visits not seen in development and compare against an agency-adjudicated reference, sampling items where runs agree.","delivery":"Delivery framing: the work is establishing a local reference set, validating grading on held-out visits, then routing grades to supervisors for coaching. Dependencies are recorded visits, expert reviewers and union or policy agreement on use. The owner is the community supervision training unit. Acceptance should require out-of-sample agreement thresholds and documented prohibition on disciplinary use."},"roleDetails":{"sales":{"customerProblem":"Community corrections agencies audit only a small fraction of supervision visits, and expert reviewers disagree.","stakeholders":["Chief probation officer","Training and quality assurance manager","Supervisors","Officers’ union","Agency counsel"],"discoveryQuestions":["Are supervision visits recorded and transcribed?","Which practice model and rubric does the agency use?","How many visits are reviewed today?","What uses of scores would be prohibited?"],"valueHypothesis":"Validated AI grading could expand coaching review coverage if it meets agreement thresholds on the agency’s own visits.","potentialEngagement":"Validation pilot on held-out local visits against an agency-adjudicated reference, coaching use only.","unsupportedClaims":["Reduced recidivism","Accuracy on unseen visits","Suitability for discipline or liberty decisions"]},"engineering":{"fit":"Agencies with recorded supervision visits and a defined core-correctional-practice rubric.","architectureIntegration":"Transcription feeds item-level grading with evidence citations into a supervisor coaching dashboard.","prerequisites":["Visit recordings and transcription","De-identification process","Local adjudicated reference set"],"constraints":["Model versions change","Transcripts miss tone and nonverbal conduct","State privacy rules on recordings"],"security":"De-identify transcripts, restrict access to supervisors, and record which model configuration produced each grade.","proofOfValue":["Proposed: measure agreement on visits not used in development","Proposed: sample unanimous-run items for hidden errors"]},"delivery":{"work":"Build a local reference, validate grading out of sample, and integrate grades into supervisor coaching.","dependencies":["Expert reviewer time","Policy on permitted uses","Recording infrastructure"],"ownership":"Community supervision training and quality assurance unit","skillsAdoption":"Supervisors trained to use cited evidence in coaching and to accept officer challenges to marks.","governanceCheckpoints":["Written prohibition on disciplinary use","Re-validation on any model or rubric change","Periodic sampled human review"],"acceptanceCriteria":["Proposed: at least 85% agreement with local reference on held-out visits","Proposed: every grade links to transcript evidence"],"risks":["Scores repurposed for discipline","Performance drops on new visits","Vendor lock-in to a model configuration"]}},"retrievedAt":"2026-10-21T02:51:39Z","enrichedAt":"2026-10-21T02:57:27Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Intended to support officer coaching; risk that scores are misused for discipline, which the authors exclude from validated use.","procurementImplications":"Interpretation: buyers should require out-of-sample validation on their own visits and contract terms prohibiting use for discipline or liberty decisions without separate validation.","operatingModelImplications":"Shifts supervision quality review from small manual audits toward broader automated review with human coaches and sampled checks.","sourceVerification":{"openedUrl":"https://osf.io/download/6abd613e284e0da501d0eb72/","referenceExcerpt":"All ten visits were used for development, so accuracy on new visits remains to be measured.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}