{"resourceId":"arxiv-race-blind-charging-bc2-202607","versions":[{"version":"external-11be82a933f111c54f047ef54b26050bd3b47dd1159a6b2f57ea6d2735213a7b","resource":{"id":"arxiv-race-blind-charging-bc2-202607","title":"Algorithm validation as policy audit: California race-blind charging redaction (bc2) reaches 96.7% mandate compliance but misses location proxies","organization":"arXiv preprint (Walia, Nudell, Chohlas-Wood)","sector":"Prosecutorial charging / academic evaluation","geography":"California prosecutors (tool used on 119,000+ 2025 cases); validation corpus of ~5,000 U.S. police reports","publishedAt":"July 27, 2026 (arXiv v1)","publicationDate":"2026-07-27","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint (cs.CY / cs.CL)","sourceUrl":"https://arxiv.org/abs/2609.13174","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["data-security","governance-procurement","knowledge-work","operating-model"],"finding":"Authors validate bc2, an open-source LLM redaction tool used to support California race-blind charging reviews in more than 119,000 real-world 2025 cases. On a corpus of nearly 5,000 U.S. police-report narratives, the latest bc2 version meets the state's redaction mandate on 96.7% of documents under a stringent document-level measure—better than earlier bc2 versions and leading open-source redaction baselines. Separately, they show the statutory proxy list misses key race predictors such as location; redacting additional proxies (as bc2 does) removes 43.1% of remaining predictive signal after mandate-compliant redaction. Interpretation: high technical compliance can still leave policy goals incomplete without broader proxy coverage and ongoing validation.","sledRelevance":"California DAs and other states copying race-blind charging need procurement and evaluation criteria that test both legal-mandate fidelity and residual race predictability—not vendor assurances alone. Open-source bc2 evidence is transferable with jurisdiction-specific legal mapping.","evidence":"Opened arXiv abstract page and HTML full text via curl (UA SLED-research-bot/1.0); extracted abstract metrics 96.7%, 43.1%, 119,000 cases, ~5,000-report corpus, and discussion framing of validation as policy audit.","architectureImplications":"LLM redaction sits upstream of charging review systems; require versioned prompts, document-level pass/fail metrics, and optional expanded-proxy modes beyond minimum statute lists.","governanceImplications":"Treat validation as ongoing policy audit: publish compliance rates, residual predictability, and when statute proxy lists need legislative/regulatory update.","securityPrivacyImplications":"Police narratives contain sensitive PII; redaction pipelines must protect data in transit/at rest and avoid leaking unredacted text via logs or vendor training.","caveats":"Preprint (submitted 2026-07-27), not peer-reviewed journal version. Authors developed bc2 (not a fully independent third-party auditor). Validation corpus is multi-jurisdiction reports, not solely California charging files. Does not measure downstream charging disparity outcomes—only redaction fidelity and residual predictability.","streamIds":["public-safety"],"roles":{"sales":"Sales framing: Race-blind charging buyers need proof the redaction stack meets the statute and still advances fairness goals. Lead with measurable mandate compliance and residual-proxy analysis, not 'AI fairness' slogans. Stakeholders are DA leadership, charging deputies, IT, and civil-rights monitors. Position open evaluation methods and expanded-proxy options as the engagement, and be explicit that statute-minimum redaction may leave location and other signals intact.","engineering":"Engineering framing: Implement document-level redaction evaluation harnesses, prompt/version pinning, and optional location-proxy scrubbers behind policy flags. Integrate with RMS/CMS export of narratives before charging screens. Security must prevent unredacted leakage. Fit is strong for offices already mandated to race-blind charge; weak if leadership wants outcome disparity claims the pipeline cannot yet measure.","delivery":"Delivery framing: Stand up redaction QA sampling, deputy training on working from redacted packets, and a quarterly validation report comparing mandate fidelity vs residual race predictability. Ownership is DA policy lead plus data science/IT. Acceptance criteria should include documented  document-level compliance thresholds and a decision log when expanding proxies beyond statute."},"roleDetails":{"sales":{"customerProblem":"Offices must automate race-proxy redaction at scale for charging review without assuming statute-minimum redaction equals race-blind decisions.","stakeholders":["District attorney / charging supervisors","Prosecutor IT / CMS owners","Civil rights / equity monitors","State regulators implementing race-blind rules"],"discoveryQuestions":["Which proxies does local law require versus optional expansions?","How is redaction quality sampled before charging?","Can location and other residual proxies be measured?","Who owns model/prompt version changes?"],"valueHypothesis":"Bounded value is high mandate fidelity plus visibility into residual race signal; not a claim that charging bias is eliminated.","potentialEngagement":"Validation sprint: measure document-level mandate compliance and residual predictability on a local narrative sample; recommend proxy-expansion policy.","unsupportedClaims":["Race-blind charging eliminates racial bias in charging","96.7% generalizes to every California office without local testing","Peer-reviewed finality of this preprint"]},"engineering":{"fit":"Fit for prosecutor CMS pipelines that can insert pre-charging redaction with evaluation hooks; unfit as a standalone 'fairness guarantee' product.","architectureIntegration":"Narrative export → LLM redaction (version-pinned) → compliance scorer → charging UI; optional expanded-proxy mode.","prerequisites":["Legal mapping of required proxies","Secure narrative datastore","Sampling/evaluation corpus","Human override when redaction fails"],"constraints":["Statute may omit location and other proxies","Author-developed tool—seek independent revalidation","Does not itself measure charging outcomes"],"security":"Encrypt narratives; restrict who sees pre-redaction text; ban vendor training on charging packets; audit access.","proofOfValue":["Document-level mandate pass rate on local sample","Measured reduction in residual race predictability after expanded proxies","Change-control records for prompt/model versions"]},"delivery":{"work":"Configure redaction service, train charging staff, implement QA sampling, and publish periodic validation-as-policy-audit reports.","dependencies":["CMS integration","Legal counsel on proxy list","Compute/API budget","Records retention rules"],"ownership":"DA policy lead owns race-blind procedure; IT owns pipeline; analysts own validation metrics.","skillsAdoption":"Deputies trained to charge from redacted packets; analysts trained on compliance vs predictability metrics.","governanceCheckpoints":["Go-live compliance threshold","Quarterly residual-signal review","Regulatory update trigger if proxies insufficient"],"acceptanceCriteria":["Agreed document-level compliance threshold met on holdout sample","Written decision on whether to expand beyond statute proxies","Incident process for failed redactions reaching chargers"],"risks":["False confidence from high compliance alone","Location proxies leaking race signal","Model drift after prompt changes without revalidation"]}},"retrievedAt":"2026-09-18T03:12:00Z","enrichedAt":"2026-09-18T03:14:30Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Charging staff must adapt to redacted packets; training should cover what signals are removed and residual uncertainty—not only tool clicks.","procurementImplications":"Require open evaluation metrics, prompt/version escrow, and independent revalidation clauses; do not accept mandate-compliance claims without local testing.","operatingModelImplications":"Make validation a recurring policy function (compliance + residual predictability), not a one-time implementation checkbox.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.13174","referenceExcerpt":"we find that the latest version of bc2 faithfully implements the legal mandate on 96.7% of narratives in our sample","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}