From the Public Safety edition of October 29, 2026
UK national Police AI lab proposes supplier benchmark testing against held-back trusted data before operational use, as Humberside describes knowledge-base-limited chatbot and staged CCTV analysis
Police AI (UK national policing AI body) / Humberside Police · Policing / AI evaluation and assurance · England and Wales, United Kingdom
- Publisher
- Emergency Services Times
- Original publication
- Emergency Services Times, Oct. 29, 2026
- Source retrieved
- 2026-10-30
- Event date
- 2026-10-29
What happened
At a panel reported by Emergency Services Times on Oct. 29, 2026, Police AI’s Lewis Lincoln Gordon said the body’s nascent AI lab is working on standards and testing so systems can be assessed in the context where they will be used; one proposal is an evaluation process in which suppliers test products against benchmarks, with trusted data held back to check performance, before operational use. Humberside Police’s Scott Dunn argued forces should invest in common foundations rather than "spending money 43 different ways." Near-term uses cited: interview transcription and summarisation, and staged retrospective CCTV analysis; real-time analytics would wait on reliable testing and safeguards. Humberside’s Bobbi chatbot is restricted to a pre-defined knowledge base and hands unanswered queries to a person. Speakers said deepfake detection tools are imperfect and data quality and cultural barriers to sharing remain major obstacles.
Why it matters
A national police body proposing held-back test data for supplier claims is a concrete assurance model that U.S. state police standards boards or consortia could adapt for multi-agency procurement.
Evidence and measured results
Opened via the WebFetch tool after curl returned HTTP 403, and read in full; trade-press write-up of a conference panel moderated by the author; no Police AI testing documentation was opened.
Limitations and uncertainty
Conference panel commentary, not a published standard or evaluation; the testing process is described as a proposal; no performance data; UK national context; the author moderated the panel.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-30; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Each police agency evaluates AI suppliers on its own, duplicating effort and relying on vendor claims.
- Stakeholders
- State police standards boards
- Regional police consortia
- Agency procurement
- Information assurance teams
- Discovery questions
- Which AI use cases do member agencies plan to buy?
- Is there curated, labelled data that could be held back for testing?
- Will suppliers accept blind evaluation?
- Value hypothesis
- Shared, held-back testing can make supplier claims verifiable once for many agencies.
- Potential engagement
- Shared evaluation protocol design for one use case.
- Unsupported claims
- That Police AI testing is operational
- Any performance outcome
Pre-sales engineering
Role takeaway
- Fit
- Fits multi-agency procurement of video analytics, transcription or chatbots.
- Architecture and integration
- Secure test environment with curated holdout data and fixed metrics; vendors submit outputs for scoring.
- Prerequisites
- Curated labelled dataset
- Data governance approval
- Agreed metrics
- Constraints
- Proposal stage only
- No published methodology
- UK context
- Security
- Holdout data may include CJI or personal data and must be access-controlled and audited.
- Proof of value
- Score two products on a small holdout set for one task.
Delivery
Role takeaway
- Work
- Choose a use case, curate holdout data, publish metrics, run supplier tests pre-award.
- Dependencies
- Member agency data sharing
- Legal review
- Vendor participation
- Ownership
- State standards body or consortium lead agency
- Skills and adoption
- Evaluation skills in the lead agency; procurement staff trained to read test results.
- Governance checkpoints
- Data-sharing approval
- Metric sign-off
- Results published to members
- Acceptance criteria
- Holdout set documented and protected
- Results shared with members
- RFPs require participation
- Risks
- Unrepresentative test data
- Benchmark gaming
- Cost of maintaining datasets
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Held-back evaluation sets require a trusted, curated police data store and a controlled testing environment separate from vendors; Humberside is building a cleaned data layer over legacy datasets.
Governance
Who approves, reviews and stays accountable for outcomes?
Centralised standards and testing aim to stop 43 forces duplicating assurance; the lab is a proposal-stage capability.
Security and privacy
What data, permissions and controls need testing?
Test data held back from suppliers must be protected; chatbot restricted to a knowledge base to avoid uncontrolled content; deepfakes threaten evidence integrity.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Transcription aims to free staff time; officers are told to question media until verified.
Procurement
What should contracts, pricing and exit terms secure?
Benchmark-and-holdout testing would make supplier claims checkable before contract, shifting evaluation cost to a shared body.
Operating model
Which teams own the service once it runs?
Staged deployment: retrospective uses first, real-time analytics only after testing and public understanding.
Publication history
- 2026-10-29Public Safety · Issue 537 resources
Stable resource ID: uk-police-ai-lab-supplier-benchmark-testing-held-back-data-proposal-20261029