{"resourceId":"raleigh-daisy-pilot-not-retained-20260923","versions":[{"version":"external-f58067d265ddccdeff38959b1b11e5bf76b90a7c427314b0b11b735a731a0a80","resource":{"id":"raleigh-daisy-pilot-not-retained-20260923","title":"Raleigh ends Daisy resident AI pilot after missing 90% correct-answer bar","organization":"City of Raleigh","sector":"Municipal resident services and customer experience innovation","geography":"Raleigh, North Carolina, United States","publishedAt":"September 23, 2026 StateTech Magazine coverage of Smart Cities Connect findings","publicationDate":"2026-09-23","eventDate":"2026-09-23","sourceName":"StateTech Magazine","sourceLabel":"Independent trade reporting of municipal conference presentation on AI pilot results","sourceUrl":"https://statetechmagazine.com/article/2026/09/smart-cities-connect-2026-raleighs-ai-pilot-shines-light-human-touch","evidenceClass":"independent-reporting","outcomeClass":"cautionary","topics":["knowledge-work","accessibility-workforce","governance-procurement","operating-model","data-security"],"finding":"On Sept. 23, 2026 StateTech reported that Raleigh tested Daisy, an AI customer-service assistant with a humanlike avatar, from April through July and ultimately decided not to keep it. Officials set a firm bar: at least 90% correct answers before spending hundreds of thousands on broader deployment. Internal testing with 25 employees showed about 30% accuracy and ~4/10 satisfaction. An outdated strategic plan naming a prior mayor/council polluted answers (“garbage in, garbage out”). By pilot end only about 5% of answers were incorrect, but many replies were non-answers lacking information, so the 90% goal was still missed. Focus groups split on whether a human face helped; residents preferred people for emotional/complex needs and were more open to AI for queue-jumping simple questions or after-hours help. Takeaway emphasized staff-facing AI for tedious work and needs-based placement rather than slapping AI on everything. Cited 2025 PayIt index: 83% of governments implementing AI vs 50% consumer comfort.","sledRelevance":"Same-day cautionary measured pilot: explicit accuracy gate, data-hygiene failure mode, avatar UX split, and decision not to procure—contrasts with go-live announcements peers may over-weight.","evidence":"StateTech Sept. 23, 2026: Daisy April–July pilot; not retained; 90% correct bar; internal ~30% accuracy / ~4/10 satisfaction; outdated strategic plan caused errors; end-state ~5% incorrect but too many “don’t have information” replies; avatar face controversial; complex/emotional questions prefer humans; staff-facing AI may offer more value.","architectureImplications":"Interpretation: Corpus currency controls and non-answer rates matter as much as factual error rates; avatar UX is a product choice with polarized acceptance.","governanceImplications":"Interpretation: Pre-commit published accuracy/usability gates and be willing to stop spend when unmet; treat shadow/outdated documents as deployment blockers.","securityPrivacyImplications":"Interpretation: Stale authoritative documents create confident wrong answers; inventory and retire superseded plans before retrieval grounding.","caveats":"Conference reporting of city officials’ briefing; exact vendor and public sample sizes beyond the 25-employee internal test are limited in the article; PayIt comfort statistic is secondary citation.","streamIds":["local-government"],"roles":{"sales":"Cities under pressure to launch resident AI still need kill criteria. Raleigh’s Daisy pilot supports discovery on accuracy gates, corpus hygiene, and avatar UX—not a case study for guaranteed deflection. Bound any engagement to a measured pilot with a pre-agreed go/no-go threshold and human paths for complex cases. Do not claim Raleigh proved production readiness; they chose not to keep the tool.","engineering":"Fit is a multilingual resident Q&A assistant with retrieval over city content—only if corpus governance is real. Architecture needs document freshness controls, non-answer telemetry, and optional non-human persona. Prerequisites: cleaned source-of-truth content and language packs. Constraints: internal 30% start accuracy; end-state still missed 90% useful-correct bar. Security: prevent superseded plans from remaining in the retrieval index.","delivery":"Run internal then public pilots against a written 90%-style gate; sample wrong and empty answers; retire bad documents; decide stop vs scale. Ownership: strategy/innovation with department content owners. Skills: staff who fix source pages the bot will mirror. Governance checkpoints: pre-pilot gate memo and end-of-pilot go/no-go. Acceptance: published accuracy and non-answer rates; no spend without meeting the gate. Risks: avatar backlash and launching despite empty answers."},"roleDetails":{"sales":{"customerProblem":"Resident service queues and language-access gaps create pressure to buy AI assistants before accuracy is proven.","stakeholders":["Strategy/innovation leads","Customer service managers","IT/content owners","Mayor/council","Residents/language-access advocates"],"discoveryQuestions":["What correct-answer and non-answer thresholds stop the procurement?","How is superseded content removed from the corpus?","Is a human avatar required, optional, or banned?","Where do complex/emotional cases hand off to people?"],"valueHypothesis":"A gated pilot can reveal whether resident AI beats empty answers and stale data—without assuming production spend is justified.","potentialEngagement":"Pilot design with written go/no-go accuracy gate, corpus cleanup, and UX testing—explicitly allowing a stop decision like Raleigh.","unsupportedClaims":["Raleigh proved a successful production chatbot","30% early accuracy is acceptable for go-live","Human avatars are universally preferred","Guaranteed call deflection or FTE savings"]},"engineering":{"fit":"Grounded multilingual resident assistant with strict corpus governance and measurable non-answer handling.","architectureIntegration":"Kiosk/web channel → retrieve approved city docs → answer or abstain → log correctness samples; optional non-human persona.","prerequisites":["Document inventory and retirement workflow","Language test sets","Accuracy sampling harness","Human handoff path"],"constraints":["Missed 90% useful-correct bar","Avatar polarization","Pilot scoped to Parks info in testing narrative","Conference-reported metrics"],"security":"Keep only current authoritative docs in the index; control who can publish to the bot corpus.","proofOfValue":["Correct-answer rate","Non-answer rate","Satisfaction","Language-access success"]},"delivery":{"work":"Clean corpus, run employee then public tests, facilitate focus groups on persona, and execute a documented stop/scale decision.","dependencies":["Content stewards","Innovation team","Vendor (if any)","Communications"],"ownership":"Strategy and Innovation owns the pilot decision; departments own source content.","skillsAdoption":"Train staff that fixing documents fixes answers; brief front-line teams on when to override.","governanceCheckpoints":["Gate memo before public pilot","Mid-pilot corpus audit","End go/no-go","Postmortem on stop decision"],"acceptanceCriteria":["Written accuracy gate exists","Stale docs removed after discovery","Stop decision documented if unmet","Human path for complex cases"],"risks":["Launching despite empty answers","Avatar controversy","Political pressure to buy anyway","Ignoring language-access failures"]}},"retrievedAt":"2026-09-24T03:22:00Z","enrichedAt":"2026-09-24T03:35:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Interpretation: Multilingual testing matters, but empty answers and avatar discomfort can still block equitable access; keep human service for complex needs.","procurementImplications":"Interpretation: Write kill criteria into pilots before six-figure deployments; pay for measured learning, not assumed production.","operatingModelImplications":"Interpretation: Prefer staff-facing AI for tedious work when resident-facing tools fail gates; redesign service halls around human problem-solving.","sourceVerification":{"openedUrl":"https://statetechmagazine.com/article/2026/09/smart-cities-connect-2026-raleighs-ai-pilot-shines-light-human-touch","referenceExcerpt":"By the end of the pilot, only about 5% of Daisy’s answers were incorrect.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}