{"resourceId":"moj-delius-semantic-search-probation-20261001","versions":[{"version":"external-351e1a2e14964a8e03d5cf7d941026fcaaa729229eab14a999617552fc805bc9","resource":{"id":"moj-delius-semantic-search-probation-20261001","title":"MoJ probation semantic search in Delius reduced search effort in user research, but measured time savings proved hard to demonstrate","organization":"Ministry of Justice / HM Prison and Probation Service (Justice Digital, Data and Science blog)","sector":"Corrections / probation","geography":"England and Wales, United Kingdom","publishedAt":"October 1, 2026","publicationDate":"2026-10-01","eventDate":null,"sourceName":"Justice Digital, Data and Science (GOV.UK blog)","sourceLabel":"MoJ Digital blog: Helping probation practitioners spend less time searching and more time supporting people","sourceUrl":"https://mojdigital.blog.gov.uk/2026/10/01/helping-probation-practitioners-spend-less-time-searching-and-more-time-supporting-people/","evidenceClass":"government-evaluation","outcomeClass":"mixed","topics":["knowledge-work","data-security","governance-procurement","accessibility-workforce"],"finding":"The Ministry of Justice's Justice Digital blog (Oct. 1, 2026) describes a hybrid keyword-plus-semantic search deployed in October 2025 in Delius, the probation case-management system that records more than 40 million contact logs a year. Analysis of the free-text search audit log showed 'search flurries' (repeated related queries) consuming meaningful staff time. Probation Data Science evaluated search models, ran an ethical evaluation under the MoJ AI and Data Science Ethics Framework, and tested bias with WEAT and product-specific testing on 30,000 probation contacts, finding no overall evidence of systematic bias. After deployment, thumbs-up/down feedback gave little context, and quantitative analysis showed only small changes in time lost to flurries, though free-text search use rose significantly; user research validated reduced effort and cognitive load in most cases with trust remaining high, while effort shifted to choosing terms and judging results. Next steps are promoting help and training and rolling the search into the Manage a Person on Probation service launched nationally in April 2026.","sledRelevance":"Corrections/probation lane: a government team's candid post-deployment evaluation of an AI-assisted knowledge tool, useful for U.S. probation and parole agencies weighing case-note search and summarization.","evidence":"Government self-evaluation blog with audit-log analysis, bias testing and user research; no published effect sizes or user-research sample sizes. Why now: published Oct. 1, not previously archived, and its finding that time savings were hard to measure bears directly on this week's AI rollout debates such as LAPD's Draft One.","architectureImplications":"Hybrid search combining exact keyword matching with semantic matching over case notes, handling misspellings, synonyms and domain acronyms such as MAPPA, embedded in an existing case-management system.","governanceImplications":"Ethics review and bias testing before deployment, plus ongoing monitoring, model a proportionate assurance path for lower-risk AI in corrections.","securityPrivacyImplications":"Semantic search over sensitive probation records makes access controls and audit logging of searches important; the blog does not detail them.","caveats":"Self-reported evaluation by the deploying department; user-research sample sizes and quantitative results are not published; time savings were not demonstrated; UK system context limits U.S. transferability; published four days before tonight's edition date.","streamIds":["public-safety"],"roles":{"sales":"Sales framing: probation agencies want faster case understanding, and MoJ's evidence shows semantic search can reduce effort, but time savings are hard to prove. Discovery should cover case-note volume, current search pain, and whether audit logs can establish a baseline. A bounded engagement is a search pilot with user research and bias testing; do not promise measurable hours saved, which MoJ could not clearly demonstrate.","engineering":"Engineering framing: the fit is hybrid search over case notes inside the existing case-management system. Engineers need domain vocabulary handling for acronyms and misspellings, search audit logs for baseline and evaluation, and bias tests on representative records. Prerequisites include access to historical contact logs under data-protection rules. Proof of value should combine query-behaviour metrics with observed user sessions, since MoJ found thumbs feedback uninformative.","delivery":"Delivery framing: plan for slow benefit realization by training staff, improving page design and help content, and monitoring adoption before judging impact. Ownership sits with the probation case-management service, with data science owning model evaluation and ethics review. Acceptance should include bias-test results and user-research findings. Risks are low adoption when interface changes are invisible to users and overclaiming savings."},"roleDetails":{"sales":{"customerProblem":"Probation officers taking over cases struggle to find relevant history across inconsistent case notes.","stakeholders":["Probation and parole leadership","Case-management system owners","Agency data science or analytics team","Line probation officers","Privacy and ethics officers","Staff representatives"],"discoveryQuestions":["How many case notes are searched and how inconsistent is terminology?","Can search audit logs establish a baseline?","What bias testing would the agency require?","Which case-management screens would host the search?"],"valueHypothesis":"Bounded: semantic search may reduce effort and cognitive load in finding case history, with time savings measured cautiously.","potentialEngagement":"Case-note search pilot with an audit-log baseline, bias testing and observational user research.","unsupportedClaims":["Quantified staff-hours saved","Improved supervision outcomes or reduced reoffending","Absence of bias for every subgroup","Direct transfer to U.S. systems without testing"]},"engineering":{"fit":"Probation, parole and community corrections agencies with large free-text case-note stores.","architectureIntegration":"Case-management contact logs to hybrid keyword-plus-semantic index with domain vocabulary to search interface to audit log for evaluation.","prerequisites":["Historical case notes accessible under data-protection rules","Search audit logging","Domain acronym and synonym lists","Bias-test datasets"],"constraints":["Sensitive personal data in case notes","Legacy case-management interface","Benefits lag adoption","UK-specific terminology"],"security":"Apply existing case access controls to search results and log every query for audit.","proofOfValue":["Reduction in repeated related queries","Observed task completion in user sessions","Bias test results on representative contacts","Adoption trend after training"]},"delivery":{"work":"Deploy hybrid search, run ethics and bias review, conduct observational user research, then train staff and improve interface guidance.","dependencies":["Case-management vendor or team","Data science capacity","Regional IT trainers","Ethics review process"],"ownership":"Probation case-management service owner, with data science owning evaluation.","skillsAdoption":"Practitioners need guidance on broad versus specific search terms and on judging result relevance.","governanceCheckpoints":["Ethics framework review before deployment","Bias test sign-off","Post-deployment user research","Expansion review before new services"],"acceptanceCriteria":["Proposed: documented bias-test results before launch","Proposed: user research showing reduced effort in most observed tasks","Proposed: audit-log baseline and six-month comparison","Proposed: help content and training delivered to all regions"],"risks":["Overclaiming time savings","Low discoverability limiting adoption","Undetected subgroup bias","Effort shifting to judging results"]}},"retrievedAt":"2026-10-06T02:54:38Z","enrichedAt":"2026-10-06T03:01:43Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Reduces cognitive load for practitioners reviewing colleagues' notes; benefits depend on training and clearer page design.","procurementImplications":"Built in-house with data science and architecture teams; buyers of commercial case-note AI should require equivalent bias testing and audit-log access.","operatingModelImplications":"Benefit realization is treated as gradual, with monitoring and user research rather than one-off ROI claims.","sourceVerification":{"openedUrl":"https://mojdigital.blog.gov.uk/2026/10/01/helping-probation-practitioners-spend-less-time-searching-and-more-time-supporting-people/","referenceExcerpt":"Quantitative analysis revealed how difficult it can be to track time savings, due to difficulties defining and interpreting user behaviour (search flurries).","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}