{"resourceId":"uk-police-ai-lab-supplier-benchmark-testing-held-back-data-proposal-20261029","versions":[{"version":"external-10f49c08ab01594855a72640951f5a604751b47fa30305a2fd9b84146a3773d3","resource":{"id":"uk-police-ai-lab-supplier-benchmark-testing-held-back-data-proposal-20261029","title":"UK national Police AI lab proposes supplier benchmark testing against held-back trusted data before operational use, as Humberside describes knowledge-base-limited chatbot and staged CCTV analysis","organization":"Police AI (UK national policing AI body) / Humberside Police","sector":"Policing / AI evaluation and assurance","geography":"England and Wales, United Kingdom","publishedAt":"Emergency Services Times, Oct. 29, 2026","publicationDate":"2026-10-29","eventDate":"2026-10-29","sourceName":"Emergency Services Times","sourceLabel":"AI conversations in policing that go beyond the headlines","sourceUrl":"https://emergencyservicestimes.com/2026/10/29/ai-conversations-in-policing-that-go-beyond-the-headlines/","evidenceClass":"independent-reporting","outcomeClass":"emerging","topics":["governance-procurement","data-security","operating-model"],"finding":"At a panel reported by Emergency Services Times on Oct. 29, 2026, Police AI’s Lewis Lincoln Gordon said the body’s nascent AI lab is working on standards and testing so systems can be assessed in the context where they will be used; one proposal is an evaluation process in which suppliers test products against benchmarks, with trusted data held back to check performance, before operational use. Humberside Police’s Scott Dunn argued forces should invest in common foundations rather than \"spending money 43 different ways.\" Near-term uses cited: interview transcription and summarisation, and staged retrospective CCTV analysis; real-time analytics would wait on reliable testing and safeguards. Humberside’s Bobbi chatbot is restricted to a pre-defined knowledge base and hands unanswered queries to a person. Speakers said deepfake detection tools are imperfect and data quality and cultural barriers to sharing remain major obstacles.","sledRelevance":"A national police body proposing held-back test data for supplier claims is a concrete assurance model that U.S. state police standards boards or consortia could adapt for multi-agency procurement.","evidence":"Opened via the WebFetch tool after curl returned HTTP 403, and read in full; trade-press write-up of a conference panel moderated by the author; no Police AI testing documentation was opened.","architectureImplications":"Held-back evaluation sets require a trusted, curated police data store and a controlled testing environment separate from vendors; Humberside is building a cleaned data layer over legacy datasets.","governanceImplications":"Centralised standards and testing aim to stop 43 forces duplicating assurance; the lab is a proposal-stage capability.","securityPrivacyImplications":"Test data held back from suppliers must be protected; chatbot restricted to a knowledge base to avoid uncontrolled content; deepfakes threaten evidence integrity.","caveats":"Conference panel commentary, not a published standard or evaluation; the testing process is described as a proposal; no performance data; UK national context; the author moderated the panel.","streamIds":["public-safety"],"roles":{"sales":"Sales framing: the UK’s national Police AI body wants suppliers to prove performance against benchmarks using trusted data the vendor never sees. For U.S. state police standards boards or regional consortia, the buyer problem is checking vendor claims once instead of in every agency. A bounded engagement could design a shared evaluation protocol for one use case, such as CCTV search. The evidence cannot support claims that this testing exists yet or that it improves outcomes; it is a proposal described on a panel.","engineering":"Engineering framing: held-back testing needs a curated, labelled police dataset, a secure environment where vendors submit models or outputs, and fixed metrics. Pre-sales should ask who would own the test data, how it is protected, and whether vendors will accept blind evaluation. A proof of value would build a small held-back set for one task, such as retrospective video search, and score two products. Avoid promising real-time analytics; the speakers placed it after testing and safeguards.","delivery":"Delivery framing: the work is setting up a shared evaluation function: choose a use case, curate and protect the holdout data, publish metrics, and run supplier tests before contract award. The operational owner is a state standards body or consortium lead agency. Acceptance criteria are proposals: holdout dataset documented and access-controlled, results shared with member agencies, and procurement documents requiring participation. Main risks: unrepresentative test data and vendors tuning to the benchmark."},"roleDetails":{"sales":{"customerProblem":"Each police agency evaluates AI suppliers on its own, duplicating effort and relying on vendor claims.","stakeholders":["State police standards boards","Regional police consortia","Agency procurement","Information assurance teams"],"discoveryQuestions":["Which AI use cases do member agencies plan to buy?","Is there curated, labelled data that could be held back for testing?","Will suppliers accept blind evaluation?"],"valueHypothesis":"Shared, held-back testing can make supplier claims verifiable once for many agencies.","potentialEngagement":"Shared evaluation protocol design for one use case.","unsupportedClaims":["That Police AI testing is operational","Any performance outcome"]},"engineering":{"fit":"Fits multi-agency procurement of video analytics, transcription or chatbots.","architectureIntegration":"Secure test environment with curated holdout data and fixed metrics; vendors submit outputs for scoring.","prerequisites":["Curated labelled dataset","Data governance approval","Agreed metrics"],"constraints":["Proposal stage only","No published methodology","UK context"],"security":"Holdout data may include CJI or personal data and must be access-controlled and audited.","proofOfValue":["Score two products on a small holdout set for one task."]},"delivery":{"work":"Choose a use case, curate holdout data, publish metrics, run supplier tests pre-award.","dependencies":["Member agency data sharing","Legal review","Vendor participation"],"ownership":"State standards body or consortium lead agency","skillsAdoption":"Evaluation skills in the lead agency; procurement staff trained to read test results.","governanceCheckpoints":["Data-sharing approval","Metric sign-off","Results published to members"],"acceptanceCriteria":["Holdout set documented and protected","Results shared with members","RFPs require participation"],"risks":["Unrepresentative test data","Benchmark gaming","Cost of maintaining datasets"]}},"retrievedAt":"2026-10-30T03:00:52Z","enrichedAt":"2026-10-30T03:06:22Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Transcription aims to free staff time; officers are told to question media until verified.","procurementImplications":"Benchmark-and-holdout testing would make supplier claims checkable before contract, shifting evaluation cost to a shared body.","operatingModelImplications":"Staged deployment: retrospective uses first, real-time analytics only after testing and public understanding.","sourceVerification":{"openedUrl":"https://emergencyservicestimes.com/2026/10/29/ai-conversations-in-policing-that-go-beyond-the-headlines/","referenceExcerpt":"One proposal is an evaluation process in which suppliers test their products against benchmarks, with trusted data held back to check performance.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}