{"resourceId":"vermont-aot-chatvt-500-weekly-users-knowledge-repo-eval-aicg-20261008","versions":[{"version":"external-2dc4b45219b5dca82d7c9bea78f120fed44d2fa0e372ab506d30eb5fc5355ae1","resource":{"id":"vermont-aot-chatvt-500-weekly-users-knowledge-repo-eval-aicg-20261008","title":"Partnership for Public Service (Oct 8): Vermont AOT reports ChatVT at ~500 weekly users, plans graded test-prompt evaluation for a 75-document transportation knowledge repository","organization":"Vermont Agency of Transportation (Rose Vath, AI & BI and performance manager); Vermont Agency of Digital Services (ChatVT)","sector":"State transportation agency knowledge work and internal generative AI assistant","geography":"Vermont statewide","publishedAt":"2026-10-08","publicationDate":"2026-10-08","eventDate":null,"sourceName":"Partnership for Public Service AI Center for Government","sourceLabel":"AI Bright Spot: Lowering costs and modeling accountability with AI in Vermont (Oct 8, 2026)","sourceUrl":"https://aicenterforgovernment.org/2026/10/08/ai-bright-spot-lowering-costs-and-modeling-accountability-with-ai-in-vermont/","evidenceClass":"public-sector-association","outcomeClass":"emerging","topics":["knowledge-work","accessibility-workforce","governance-procurement","operating-model"],"finding":"In an Oct 8 interview, Vermont AOT's Rose Vath says ChatVT, the state's in-house employee assistant, has about 500 weekly users and was designed for cost minimization as a 'safety hatch' if token costs of public tools like ChatGPT, Claude and Gemini become burdensome; Vermont policy allows public AI tools with public data. AOT is only recently adopting Microsoft AI Hub (Power Automate) and Copilot Studio, deliberately preferring single AI steps inside deterministic workflows over agentic solutions. AOT is developing a knowledge repository over about 75 procedures, manuals and handbooks, with a planned evaluation using a test-prompt library graded by test users for completeness and accuracy. Vath runs a monthly 'On Ramp for AI at AOT' workshop.","sledRelevance":"Fresh adoption data point for a state-built assistant (up from nearly 200 users at end of 2024 per ADS's annual report) and an agency-level evaluation plan for a RAG-style knowledge tool, complementing Vermont ADS's Anthropic oversight story with ordinary knowledge-work use.","evidence":"Opened the Partnership for Public Service AI Center for Government blog post (article:published_time 2026-10-08) via curl and extracted full text.","architectureImplications":"Mixed estate: in-house ChatVT, Microsoft Enterprise AI Hub/Copilot Studio, and policy-permitted public tools on public data; knowledge repository implies retrieval over ~75 agency documents.","governanceImplications":"Risk posture favors bounded AI steps within deterministic workflows over agentic automation; evaluation design is still a plan, not results.","securityPrivacyImplications":"Public-tool use is restricted to public data; the article gives no detail on ChatVT data controls or logging.","caveats":"Interview-based association blog; usage figure is self-reported, with no denominator, cost data or measured savings. Knowledge repository is in development and its evaluation not yet run. Celebratory 'Bright Spot' framing.","streamIds":["state-government"],"roles":{"sales":"Vermont shows a state hedging commercial AI costs with an in-house assistant now at roughly 500 weekly users, while its transportation agency adds Copilot Studio slowly and avoids agentic workflows. Lead with knowledge-repository evaluation help and cost transparency, not agent automation. Discovery should probe total cost per user across ChatVT and commercial tools. Do not claim savings: the source reports usage and intent, not measured cost reduction or accuracy results. This is interpretation from one interview.","engineering":"The architecture is a mixed estate: in-house ChatVT, Microsoft Copilot Studio and Power Automate AI Hub, and public tools limited to public data. The AOT repository over about 75 procedures is a classic retrieval case; propose a graded test-prompt harness, source citation, and document freshness ownership. Keep AI as a single step inside deterministic flows, matching Vath's stated risk posture. Validate grounding and accuracy before broad rollout; no evaluation results exist yet.","delivery":"Delivery work is an evaluation-first knowledge repository: build a prompt library, recruit test users, score completeness and accuracy, and tie launch to thresholds. Pair it with the agency's monthly workshop so staff know approved tools and escalation paths. Owners need to keep the 75 source documents current. Track ChatVT adoption beyond weekly users, such as task types and errors. Acceptance criteria below are proposals, since the source reports plans rather than outcomes."},"roleDetails":{"sales":{"customerProblem":"Agency procedures are scattered across ~75 documents and commercial AI token costs may grow.","stakeholders":["Agency BI/AI manager","State chief data and AI officer","Agency project managers","ADS ChatVT team"],"discoveryQuestions":["What does ChatVT cost per weekly user versus commercial licenses?","Who owns accuracy of the 75 source documents?","What accuracy threshold gates repository launch?","Which workflows may include an AI step?"],"valueHypothesis":"A graded evaluation can show whether a retrieval tool reliably answers procedure questions for new staff.","potentialEngagement":"Short evaluation-harness build for one agency knowledge repository.","unsupportedClaims":["Measured cost savings from ChatVT","Accuracy of the knowledge repository","Statewide productivity gains"]},"engineering":{"fit":"Fits retrieval over a bounded agency document set inside existing Microsoft and in-house tooling.","architectureIntegration":"Index ~75 AOT documents for ChatVT or Copilot Studio retrieval; embed AI as one step in deterministic Power Automate flows.","prerequisites":["Document inventory and owners","Test-prompt library","Approved platform choice"],"constraints":["Public tools only with public data","Agentic designs disfavored","Vermont policy specific"],"security":"Keep non-public procedures within state-controlled ChatVT or enterprise tenant; log queries.","proofOfValue":["Graded accuracy and completeness on a test-prompt set","Citation to source document per answer","Comparison to manual lookup time"]},"delivery":{"work":"Build, evaluate and launch an AOT procedure knowledge repository with training.","dependencies":["Document owners across AOT teams","ADS platform support","Test-user availability"],"ownership":"AOT business intelligence unit with ADS support.","skillsAdoption":"Monthly On Ramp for AI workshop plus scenario walkthroughs.","governanceCheckpoints":["Evaluation plan approved before build","Threshold review before launch","Periodic document freshness review"],"acceptanceCriteria":["Proposed: >=90% of graded test prompts rated complete and accurate","Proposed: every answer cites a source document","Proposed: document owners named for all 75 sources"],"risks":["Stale source documents produce confident wrong answers","Usage metric mistaken for value","Cost hedge unproven"]}},"retrievedAt":"2026-10-09T22:22:35-05:00","enrichedAt":"2026-10-09T22:22:35-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Monthly agency workshop and accountability framing ('if something goes wrong, they will have support') position training as the adoption lever.","procurementImplications":"ChatVT framed as a hedge against per-token/license costs of commercial tools; no cost comparison published.","operatingModelImplications":"Agency BI unit selects tools and use cases under central ADS AI leadership.","sourceVerification":{"responseId":"curl-aicg-vt-aot-20261009","openedUrl":"https://aicenterforgovernment.org/2026/10/08/ai-bright-spot-lowering-costs-and-modeling-accountability-with-ai-in-vermont/","referenceExcerpt":"Across the state government, ChatVT now has about 500 weekly users, Vath said.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}