{"resourceId":"graides-westminster-ai-assurance-20260618","versions":[{"version":"external-236f8b0837e2934a8ebb671cbb6ae9dc00f72355e59d3c191d05a01c85e00813","resource":{"id":"graides-westminster-ai-assurance-20260618","title":"GRAIDES: open schema for generative AI evaluation and assurance in local government (Westminster City Council)","organization":"Westminster City Council AI Innovation Lab (arXiv preprint)","sector":"Local government / academic methods","geography":"United Kingdom (Westminster City Council); transferable methods","publishedAt":"18 June 2026 (arXiv 2606.20963v1)","publicationDate":"2026-06-18","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic research preprint","sourceUrl":"https://arxiv.org/pdf/2606.20963","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["governance-procurement","data-security","operating-model","developers-agents"],"finding":"Authors from Westminster City Council’s AI Innovation Lab propose GRAIDES, a lightweight open-source data model to centralize generative-AI observability across vendors—linking interactions, model/prompt configurations, and evaluation artefacts for reproducible assurance. They argue fragmented procurement and inconsistent metrics undermine efficiency and Responsible AI adherence. Illustrative case studies from Westminster’s AI catalogue focus on measuring human–model alignment and detecting systematic disagreement between evaluators; guidance spans rapid human-approval workflows to deeper text/classification evaluations.","sledRelevance":"Why-now: as U.S. localities fund permitting/resident AI (e.g., WA Sept. 24 grants), they still lack shared evaluation plumbing. GRAIDES offers a concrete assurance schema smaller governments can adapt instead of inventory-only policies.","evidence":"Opened arXiv abs/PDF 2606.20963v1 (18 Jun 2026). Methods/architecture paper with Westminster catalogue case studies; not a multi-city controlled trial. Transfer to U.S. SLED requires mapping to local records/privacy rules.","architectureImplications":"Treat evaluation logs as first-class data: relational store (e.g., PostgreSQL/Snowflake) linking prompts, configs, human ratings, and model outputs across vendors for longitudinal comparison.","governanceImplications":"Moves governance from policy PDFs to auditable evaluation datasets; supports reproducible benchmarking and disagreement analysis among human evaluators before scaling GenAI services.","securityPrivacyImplications":"Centralizing interaction logs concentrates sensitive resident/case content—requires access control, redaction, and retention aligned to local records law; schema itself does not solve privacy.","caveats":"Preprint; illustrative Westminster cases, not a multi-city outcome trial. Open-source schema adoption cost and SME evaluation burden may challenge small U.S. cities. U.K. context.","streamIds":["local-government"],"roles":{"sales":"Municipal buyers drowning in vendor GenAI pilots need a way to compare performance and safety evidence—not another chatbot. GRAIDES, from Westminster’s AI lab, frames assurance as a shared data model linking prompts, configs, and human evaluations across tools. Position it as evaluation infrastructure beside any resident or permitting assistant. Bound the pitch to better auditability and human–model alignment measurement; this paper does not prove service-level outcome gains for U.S. cities.","engineering":"Fit as an observability/evaluation layer over heterogeneous GenAI vendors, not a replacement model. Implement schema tables for interactions, configurations, and evaluation artefacts; pipe production traces and golden-set runs into one store. Prerequisites: logging hooks in each pilot, SME raters, and a privacy review for stored prompts/outputs. Constraints: U.K.-origin design; SME time; multi-vendor idiosyncrasies. Proof of value is reproducible cross-system comparisons and detected rater disagreement—not chatbot CSAT alone.","delivery":"Stand up a minimal GRAIDES store, define rubrics for one high-traffic use case, run parallel human and automated evaluations, and report alignment gaps to an AI governance board. Dependencies include the data platform, departmental SMEs, and vendor log export. Ownership: AI/innovation lead with the records and privacy officer. Risks: schema theater without SME capacity, over-centralizing sensitive logs, and assuming U.K. patterns transfer unchanged to U.S. cities."},"roleDetails":{"sales":{"customerProblem":"Cities run fragmented GenAI pilots with inconsistent metrics, so leaders cannot compare safety/performance across vendors or departments.","stakeholders":["CIO/CDO","AI governance board","Service department SMEs","Privacy/records officer","Internal audit"],"discoveryQuestions":["Where are prompt/output logs stored today per pilot?","Which use cases have human evaluation rubrics?","Can vendors export interaction traces?","Who adjudicates rater disagreement?"],"valueHypothesis":"A shared evaluation schema can make GenAI assurance comparable across vendors—Westminster’s catalogue cases illustrate alignment measurement—without claiming automatic service improvements.","potentialEngagement":"Five-week assurance-data pilot: map two GenAI tools into a GRAIDES-like schema and deliver one cross-system evaluation report.","unsupportedClaims":["Guaranteed bias elimination","Turnkey U.S. compliance from this schema","That Westminster results generalize to all municipalities"]},"engineering":{"fit":"Cross-vendor GenAI observability and evaluation data modeling for local-government catalogues.","architectureIntegration":"Central DB/lakehouse ingesting vendor traces + human ratings; optional Snowflake/Postgres DDL; dashboards for alignment/disagreement.","prerequisites":["Log export from each GenAI tool","Rubric definitions","Identity-aware access to evaluation store","Retention/redaction policy"],"constraints":["SME rating capacity","Vendor API limits","Sensitive content in prompts","Preprint maturity"],"security":"Encrypt evaluation corpus; minimize PII in stored prompts; segregate duties between operators and auditors; document model/prompt change history.","proofOfValue":["Two systems comparable on same rubric","Rater disagreement detected and triaged","Time to assemble audit pack reduced","Reproducible re-run of evaluation battery"]},"delivery":{"work":"Deploy schema, instrument two pilots, train SME raters, publish first alignment report to governance board.","dependencies":["Data platform team","Vendor cooperation","Legal/privacy approval","Service SMEs"],"ownership":"AI Innovation / CDO owns schema; departments own ratings for their systems.","skillsAdoption":"SMEs learn structured evaluation; engineers learn to treat prompts/configs as governed artifacts.","governanceCheckpoints":["Privacy review before log centralization","Rubric sign-off","Quarterly cross-system review","Sunset criteria for failing tools"],"acceptanceCriteria":["Schema populated for ≥2 production GenAI uses","Documented human–model alignment metrics","Disagreement protocol exercised once","Audit trail for prompt/model changes"],"risks":["Evaluation burden stalls adoption","Schema without decisions","Cross-border transfer assumptions"]}},"retrievedAt":"2026-09-25T22:16:00-05:00","enrichedAt":"2026-09-25T22:20:53.522138-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Structured evaluation increases SME workload; without training, smaller councils may skip assurance. Clear rubrics can also reduce informal/uneven quality judgments that hurt residents.","procurementImplications":"Require vendors to export interaction/evaluation-ready logs and accept third-party rubric scoring; score bids on observability, not only features.","operatingModelImplications":"Creates a central assurance function that compares departmental GenAI, changing operating model from siloed pilots to catalogue-level governance.","sourceVerification":{"openedUrl":"https://arxiv.org/pdf/2606.20963","referenceExcerpt":"Illustrative case study results are reported from Westminster City Council's AI catalogue with a focus on measuring human-model alignment.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}