{"resourceId":"california-engaged-ai-impact-report-ai-assisted-analysis-20261013","versions":[{"version":"external-6f95cfcc196138cb1ea3869ed84a03ee2a9c7f11f773733e6ed67f15c7fa970e","resource":{"id":"california-engaged-ai-impact-report-ai-assisted-analysis-20261013","title":"California's Engaged California AI-impact report used AI for transcription, summaries and tagging inside the state's secure environment, with staff-only theme selection and citation checks","organization":"California Office of Data and Innovation (ODI) / Government Operations Agency, Engaged California","sector":"State digital government / public engagement and policy analysis","geography":"California statewide","publishedAt":"2026-10-13","publicationDate":"2026-10-13","eventDate":"2026-10-13","sourceName":"Engaged California (State of California)","sourceLabel":"Engaged California, 'How we did this work' methods page for the Impact of AI on work report (announced by the Governor's office Oct 13, 2026)","sourceUrl":"https://engaged.ca.gov/ai-impact/report-details/","evidenceClass":"government-evaluation","outcomeClass":"emerging","topics":["knowledge-work","data-security","governance-procurement","operating-model"],"finding":"California's Office of Data and Innovation published the first statewide Engaged California report (announced Oct 13, 2026): 2,702 survey respondents, 937 invited by demographic sortition, 103 attendees across 14 two-hour Zoom discussions, yielding 26 ranked policy ideas. The methods page says staff used AI to transcribe discussion audio, produce high-level summaries of discussions and survey responses, tag conversation segments by idea, and assess respondents' outlook on AI to balance viewpoints. Stated controls: all AI work ran inside the state's secure data environment, nothing was shared with AI vendors or used for training, AI did not select invitees or write or edit participant words, staff chose highlighted themes without AI, AI analysis had to give exact citations that staff checked, and every quote was validated against the transcript. In testing, AI summaries were compared with human notetakers' notes and 'covered the same ground without inventing content'. A May 2026 executive order directs agencies to use the findings where they can.","sledRelevance":"A rare state publication that documents exactly where AI was and was not allowed in a government analysis workflow, including data residency, vendor no-sharing and human-only judgment points. Transferable to state agencies using AI on public comments, listening sessions, rulemaking dockets, or legislative testimony. Relevance is limited by scale (a few thousand inputs) and by the absence of quantified accuracy results.","evidence":"Opened the original Engaged California methods page and companion report page over HTTPS (curl, HTML text extraction) and the Governor's Oct 13 release (WebFetch). Participation counts (2,702 / 937 / 103 / 14 / 26) come from the report page; AI-use controls and the human-notetaker comparison come from the methods page. An independent May 2026 RAND/Georgetown/Princeton preprint on LLM coding of 1,260 USDA public comments was opened as contrasting scrutiny; it is not evidence about California's system.","architectureImplications":"Pattern: speech-to-text plus LLM summarization and tagging running inside a state-controlled secure data environment, with outputs required to carry exact citations to transcript spans so staff can verify them. Requires transcript storage, segment-level citation linking and a review interface; the report does not name the models, hosting, or tooling.","governanceImplications":"Codifies human-only decision points (invitee selection by lottery, theme selection by staff, quotes validated against transcripts) and publishes them, which is a usable template for agency AI-use disclosure. Governance gap: no published error rate, reviewer workload, or criteria for when AI tags were overridden. Using AI to 'assess respondents' outlook on AI' for balancing is a classification step that may warrant its own validation and disclosure.","securityPrivacyImplications":"Stated controls: processing inside the state's secure data environment, no sharing of transcripts or survey responses with AI vendors, no training on content. These are self-attested; no audit, contract terms or architecture detail is published.","caveats":"Self-reported methods by the program operator; the comparison against human notetakers is described qualitatively with no sample size, metric or error count. Turnout from invitation to attendance was about 11% (103 of 937), so discussion representativeness is limited despite sortition. Independent research (opened preprint) finds different LLMs can categorize the same public comments materially differently, which a single-model, notes-overlap test would not detect. The report does not state publication date on-page; the date is taken from the Governor's Oct 13 announcement.","streamIds":["state-government"],"roles":{"sales":"Interpretation: the selling point is controlled AI for public-input analysis, not automation of judgment. California's office documents a secure-environment, citation-checked workflow for transcripts and surveys, giving agencies a peer reference when they face large comment dockets or listening sessions. Position a bounded assessment of an agency's public-input workflow against these controls. Avoid promising accuracy or time savings: California published no measured error rate or hours saved, and its human comparison was qualitative.","engineering":"Interpretation: fit is a transcription, summarization and tagging pipeline inside the agency's own secure environment, where every AI statement must cite a transcript or response span that a reviewer checks. Integrate with existing survey and meeting platforms and records retention. Prerequisites include data residency, vendor no-training terms and a review UI. Proof of value should go beyond overlap with human notes: measure tag agreement with human coders, test multiple models for divergence, and log overrides, because independent research shows single-model checks miss interpretive disagreement.","delivery":"Interpretation: delivery is a governed analysis service owned by the digital or policy office. Work includes setting human-only decision points, citation-verification steps, quote validation and a published methods note for each engagement. Train analysts to verify citations and handle ambiguous inputs. Propose acceptance criteria such as citation verification rates and human-AI tag agreement rather than claiming results California has not reported. Key risks are under-representative turnout, single-model framing bias, and unreported reviewer workload."},"roleDetails":{"sales":{"customerProblem":"Agencies receive large volumes of public comments, testimony and listening-session input and lack staff to transcribe, synthesize and tag it quickly while keeping data secure and decisions human-owned.","stakeholders":["State chief data or innovation officer","Agency policy and rulemaking leads","Public engagement or communications office","CISO and privacy officer","Records and legal counsel"],"discoveryQuestions":["How much public input do you process per rulemaking or engagement, and how long does synthesis take today?","Can AI processing run inside your own secure environment, and do current vendor terms prohibit retention and training?","Which steps must stay human-only, such as theme selection or quote use, and is that written down?","How do you currently show the public how input was analyzed?"],"valueHypothesis":"A citation-checked, in-environment AI workflow may shorten transcription and first-pass synthesis for public input while keeping theme selection and quotes human-validated; size of savings is unproven.","potentialEngagement":"Fixed-scope review of one agency's public-input analysis workflow against California-style controls, with a pilot design and a published methods-disclosure template.","unsupportedClaims":["Any specific accuracy rate or error rate for AI summaries","Hours or cost saved by AI-assisted analysis","That AI-assisted synthesis improves representativeness of public input","That California's approach is vendor- or model-specific"]},"engineering":{"fit":"Agencies with recorded meetings, open-text surveys or comment dockets and an existing secure data environment or enterprise AI tenant.","architectureIntegration":"Ingest audio and survey text into the secure environment, transcribe, run LLM summarization and segment tagging that returns exact citations, and present outputs in a reviewer interface linked to source spans; keep quotes pulled only from verified transcripts.","prerequisites":["Speech-to-text and LLM services hosted in-tenant or in a state-controlled environment","Contract terms barring vendor retention and model training","Segment-level citation linking between outputs and source text","Records-retention mapping for transcripts and AI outputs"],"constraints":["California disclosed no model, hosting or tooling, so architecture must be designed independently","Public-records and retention laws differ by state and may cover AI intermediate outputs","Small discussion samples limit statistical claims from tagged themes"],"security":"Process all transcripts and responses inside the agency's controlled environment, block vendor access and training, restrict reviewer access by role, and log AI prompts and outputs for audit.","proofOfValue":["Measure agreement between AI tags and independent human coders on a stratified sample","Run two or more models on the same input and flag divergent categorizations for human review","Track the share of AI citations that staff verify as accurate","Record reviewer time per hour of discussion to test any efficiency claim"]},"delivery":{"work":"Define the AI-assisted analysis workflow, human-only decision points, citation-verification and quote-validation steps, and a public methods disclosure, then run one engagement as a pilot.","dependencies":["Secure environment and approved AI services","Executive or agency sponsor who will use the findings","Facilitation and recruitment capacity for representative participation","Legal review of records and privacy obligations"],"ownership":"State or agency digital/innovation office running the engagement, with the policy office owning interpretation and use of findings.","skillsAdoption":"Analysts are trained to verify AI citations, recode ambiguous segments and document overrides; facilitators learn the consent and disclosure language for AI processing.","governanceCheckpoints":["Approval of AI use and data handling before recordings start","Staff sign-off on themes selected without AI assistance","Quote validation against transcripts before publication","Publication of a methods note describing AI use and limits"],"acceptanceCriteria":["Proposed: at least 95% of sampled AI citations verified as accurate by staff","Proposed: human-AI tag agreement reported on a stratified sample of at least 40 segments","Proposed: zero participant quotes published without transcript validation","Proposed: methods disclosure published with the report"],"risks":["Low turnout from invitation to attendance skews whose views are summarized","Single-model framing can hide interpretive disagreement","Reviewer workload may offset time saved and goes unmeasured","Self-attested data controls without independent audit"]}},"retrievedAt":"2026-10-14T22:16:54-05:00","enrichedAt":"2026-10-14T22:19:28-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Staff remained the analysts; AI reduced transcription and first-pass synthesis effort, but reviewer time is not reported. The project's subject is AI and work, and participants' top ideas included human final judgment on life-affecting decisions, which aligns with the program's own human-review design.","procurementImplications":"Agencies buying AI for public-input analysis can require vendor no-retention and no-training terms, in-tenant or state-environment processing, citation-to-source output, and exportable audit logs. The report names academic and civil-society design partners (Carnegie California and others) but no AI vendor or contract vehicle.","operatingModelImplications":"Shows an operating model where a central digital office runs deliberation, AI-assisted synthesis and publication, then feeds results to agencies under an executive order. Sustaining it requires facilitation staff, analysts trained to verify AI citations, and a standing publication practice for methods.","sourceVerification":{"responseId":"curl-engaged-ca-ai-impact-report-details-20261014","openedUrl":"https://engaged.ca.gov/ai-impact/report-details/","referenceExcerpt":"All AI work ran inside the state's secure data environment. We never shared transcripts or survey responses with AI vendors.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}