From the Public Safety edition of September 17, 2026
Algorithm validation as policy audit: California race-blind charging redaction (bc2) reaches 96.7% mandate compliance but misses location proxies
arXiv preprint (Walia, Nudell, Chohlas-Wood) · Prosecutorial charging / academic evaluation · California prosecutors (tool used on 119,000+ 2025 cases); validation corpus of ~5,000 U.S. police reports
- Publisher
- arXiv
- Original publication
- July 27, 2026 (arXiv v1)
- Source retrieved
- 2026-09-18
What happened
Authors validate bc2, an open-source LLM redaction tool used to support California race-blind charging reviews in more than 119,000 real-world 2025 cases. On a corpus of nearly 5,000 U.S. police-report narratives, the latest bc2 version meets the state's redaction mandate on 96.7% of documents under a stringent document-level measure—better than earlier bc2 versions and leading open-source redaction baselines. Separately, they show the statutory proxy list misses key race predictors such as location; redacting additional proxies (as bc2 does) removes 43.1% of remaining predictive signal after mandate-compliant redaction. Interpretation: high technical compliance can still leave policy goals incomplete without broader proxy coverage and ongoing validation.
Why it matters
California DAs and other states copying race-blind charging need procurement and evaluation criteria that test both legal-mandate fidelity and residual race predictability—not vendor assurances alone. Open-source bc2 evidence is transferable with jurisdiction-specific legal mapping.
Evidence and measured results
Opened arXiv abstract page and HTML full text via curl (UA SLED-research-bot/1.0); extracted abstract metrics 96.7%, 43.1%, 119,000 cases, ~5,000-report corpus, and discussion framing of validation as policy audit.
Limitations and uncertainty
Preprint (submitted 2026-07-27), not peer-reviewed journal version. Authors developed bc2 (not a fully independent third-party auditor). Validation corpus is multi-jurisdiction reports, not solely California charging files. Does not measure downstream charging disparity outcomes—only redaction fidelity and residual predictability.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-18; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Offices must automate race-proxy redaction at scale for charging review without assuming statute-minimum redaction equals race-blind decisions.
- Stakeholders
- District attorney / charging supervisors
- Prosecutor IT / CMS owners
- Civil rights / equity monitors
- State regulators implementing race-blind rules
- Discovery questions
- Which proxies does local law require versus optional expansions?
- How is redaction quality sampled before charging?
- Can location and other residual proxies be measured?
- Who owns model/prompt version changes?
- Value hypothesis
- Bounded value is high mandate fidelity plus visibility into residual race signal; not a claim that charging bias is eliminated.
- Potential engagement
- Validation sprint: measure document-level mandate compliance and residual predictability on a local narrative sample; recommend proxy-expansion policy.
- Unsupported claims
- Race-blind charging eliminates racial bias in charging
- 96.7% generalizes to every California office without local testing
- Peer-reviewed finality of this preprint
Pre-sales engineering
Role takeaway
- Fit
- Fit for prosecutor CMS pipelines that can insert pre-charging redaction with evaluation hooks; unfit as a standalone 'fairness guarantee' product.
- Architecture and integration
- Narrative export → LLM redaction (version-pinned) → compliance scorer → charging UI; optional expanded-proxy mode.
- Prerequisites
- Legal mapping of required proxies
- Secure narrative datastore
- Sampling/evaluation corpus
- Human override when redaction fails
- Constraints
- Statute may omit location and other proxies
- Author-developed tool—seek independent revalidation
- Does not itself measure charging outcomes
- Security
- Encrypt narratives; restrict who sees pre-redaction text; ban vendor training on charging packets; audit access.
- Proof of value
- Document-level mandate pass rate on local sample
- Measured reduction in residual race predictability after expanded proxies
- Change-control records for prompt/model versions
Delivery
Role takeaway
- Work
- Configure redaction service, train charging staff, implement QA sampling, and publish periodic validation-as-policy-audit reports.
- Dependencies
- CMS integration
- Legal counsel on proxy list
- Compute/API budget
- Records retention rules
- Ownership
- DA policy lead owns race-blind procedure; IT owns pipeline; analysts own validation metrics.
- Skills and adoption
- Deputies trained to charge from redacted packets; analysts trained on compliance vs predictability metrics.
- Governance checkpoints
- Go-live compliance threshold
- Quarterly residual-signal review
- Regulatory update trigger if proxies insufficient
- Acceptance criteria
- Agreed document-level compliance threshold met on holdout sample
- Written decision on whether to expand beyond statute proxies
- Incident process for failed redactions reaching chargers
- Risks
- False confidence from high compliance alone
- Location proxies leaking race signal
- Model drift after prompt changes without revalidation
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
LLM redaction sits upstream of charging review systems; require versioned prompts, document-level pass/fail metrics, and optional expanded-proxy modes beyond minimum statute lists.
Governance
Who approves, reviews and stays accountable for outcomes?
Treat validation as ongoing policy audit: publish compliance rates, residual predictability, and when statute proxy lists need legislative/regulatory update.
Security and privacy
What data, permissions and controls need testing?
Police narratives contain sensitive PII; redaction pipelines must protect data in transit/at rest and avoid leaking unredacted text via logs or vendor training.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Charging staff must adapt to redacted packets; training should cover what signals are removed and residual uncertainty—not only tool clicks.
Procurement
What should contracts, pricing and exit terms secure?
Require open evaluation metrics, prompt/version escrow, and independent revalidation clauses; do not accept mandate-compliance claims without local testing.
Operating model
Which teams own the service once it runs?
Make validation a recurring policy function (compliance + residual predictability), not a one-time implementation checkbox.
Publication history
- 2026-09-17Public Safety · Issue 114 resources
Stable resource ID: arxiv-race-blind-charging-bc2-202607