{"resourceId":"arxiv-northwestern-google-ai-overviews-health-grounding-audit-preprint-20261029","versions":[{"version":"external-1837bf054690069b880664ab041274060fdee8858b99039688650be523237a6c","resource":{"id":"arxiv-northwestern-google-ai-overviews-health-grounding-audit-preprint-20261029","title":"Northwestern preprint: Google AI Overviews (Gemini) on 3,611 health questions cite fewer government/academic sources, ~57% of cited statements supported, 22% include anticipated harmful claims","organization":"Northwestern University (Agnoli, Reilly, Barna, Horvát)","sector":"Public health information; health & human services communications","geography":"United States (U.S. English locale searches)","publishedAt":"2026-10-29 (arXiv v1)","publicationDate":"2026-10-29","eventDate":null,"sourceName":"arXiv","sourceLabel":"Preprint, not peer reviewed","sourceUrl":"https://arxiv.org/abs/2610.39339","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","data-security","governance-procurement"],"finding":"Source finding (Preprint, not peer reviewed): researchers queried 3,611 HealthBench health questions via SerpAPI (Sept. 16–18, 2026). AI Overviews appeared for 96.4% of questions; only 44.4% of the traditional top-five results were cited by the AI Overview, which over-represented commercial publishers versus academic and government sources (e.g., NHS share 1.5% vs 3.1%). After correcting for classifier error, an estimated 56.7% of cited statements were supported by their cited page; 20.8% of statements were uncited factual claims. Against physician rubrics, AI Overviews earned 38.2% of points (accuracy 54.2%, completeness 21.6%), and 22.2% of those with an anticipated harmful claim contained it. Accuracy was unrelated to grounding.","sledRelevance":"Public health departments and HHS agencies depend on residents finding authoritative guidance; this study indicates Gemini-powered AI Overviews may displace government health sources in search answers. Why now: it complements the Nov. 5 Princeton voting-information audit in this edition, giving a second, larger independent measurement of Google's AI answers on public-interest information, and it is not yet in the archive. Cross-tagged to health-human-services because the measured domain is health information.","evidence":"Preprint with detailed methods: 3,611 questions, 3,474 usable overviews, 24,060 statements; JAMA and HONcode source-quality coding (automated, kappa .68 vs humans); HallOumi grounding classifier (kappa .61, corrected with Rogan–Gladen); GPT-4.1 grading against HealthBench physician rubrics (macro-F1 .707 vs physicians). Data and code on GitHub/Zenodo. Baseline is Google's own top-five traditional results.","architectureImplications":"Interpretation: for agency-built Gemini/Vertex AI health assistants, the finding that citations did not predict accuracy argues for grounding restricted to agency-approved sources plus independent claim verification, rather than trusting the presence of citations.","governanceImplications":"Supports agency policies that warn residents to verify AI-generated health answers and that require source-restriction and human review for any agency-deployed health assistant. Authors recommend Google prefer academic/government domains and suppress AI Overviews for complex or risky questions; those are recommendations, not Google commitments.","securityPrivacyImplications":"No personal data in the study; the risk is harmful or ungrounded health guidance reaching residents.","caveats":"Preprint, not peer reviewed. HealthBench questions are longer than typical searches; automated coders and an LLM grader were used; three-day scrape; AI Overviews change frequently. Measures public Google Search, not an agency-configured Gemini deployment, so transferability to grounded enterprise builds is limited.","streamIds":["google","health-human-services"],"roles":{"sales":"Sales framing: public health communicators need to know that residents' first answer may now be a Gemini-generated AI Overview that under-cites government sources. This preprint gives measured gaps (about 57% of cited statements supported; 22% with anticipated harmful claims), useful for discovery on whether an agency needs a source-restricted assistant or better content structure. Position carefully: it is evidence of risk in public search, not proof that any Google enterprise product performs this way.","engineering":"Engineering framing: the key technical finding is that citations did not predict accuracy, so citation display alone is not a control. For a Vertex AI or Gemini-based health assistant, restrict retrieval to approved agency and CDC/NIH content, add automated claim-to-source verification, and evaluate against clinician-written rubrics. The study's open pipeline could be adapted as a test harness. Limits: different query distribution and a public-search configuration.","delivery":"Delivery framing: any health-information AI pilot should include a clinical review loop and a measurable grounding target before launch. A team could adapt the study's open pipeline to an agency's top resident questions, measure supported-statement rates, and gate release on clinician review. Ownership belongs with the public health communications or medical director's office. Interpretation only: these steps are proposed controls, not results observed in an agency."},"roleDetails":{"sales":{"customerProblem":"Residents receive AI-generated health answers that may omit or contradict agency guidance and under-cite government sources.","stakeholders":["Public health communications directors","Medical directors/health officers","HHS agency CIOs","Medicaid member-services leads"],"discoveryQuestions":["Do you know what AI search tells residents about your top health topics?","Would a source-restricted assistant be acceptable to your medical director?","What clinical review capacity exists for AI outputs?","Is your guidance published in structured, citable form?"],"valueHypothesis":"Interpretation: a bounded evaluation could quantify how often AI answers misstate agency guidance on priority topics; the source does not show any remediation effect.","potentialEngagement":"Bounded: adapt the published open pipeline to 100–200 agency priority questions and report grounding and rubric scores.","unsupportedClaims":["That agency-deployed Gemini assistants share these error rates","That Google will change AI Overviews sourcing","Any health-outcome impact"]},"engineering":{"fit":"Fit as an evaluation method and risk input for any Gemini/Vertex AI health-information use case.","architectureIntegration":"Retrieval restricted to approved agency/federal health content; claim-level verification against retrieved passages; rubric-based grading with clinician sampling.","prerequisites":["Curated corpus of approved health guidance","Clinician-written rubrics for priority questions","Logging of answers and citations for review"],"constraints":["Public AI Overviews are not configurable by agencies","Study used English, U.S. locale only","LLM grader and classifiers have known error rates"],"security":"No PHI needed for evaluation; production assistants touching member data need HIPAA-aligned controls beyond this study.","proofOfValue":["Measure supported-statement rate on agency questions","Test whether uncited claims appear","Compare with a source-restricted configuration"]},"delivery":{"work":"Run a grounding and clinical-accuracy evaluation before any health-information assistant goes live.","dependencies":["Medical director sign-off","Content owners for guidance pages","Evaluation tooling (open pipeline)"],"ownership":"Public health communications lead with the health officer's clinical review.","skillsAdoption":"Staff need training to review AI outputs against rubrics and recognize omission errors, which dominated lost points in the study.","governanceCheckpoints":["Approve source corpus","Clinical review of sample outputs","Go/no-go on grounding threshold"],"acceptanceCriteria":["Proposed: supported-statement rate above an agency-set threshold on priority questions","Proposed: zero anticipated-harmful claims in a reviewed sample","Proposed: emergency-referral questions route to care in all reviewed cases"],"risks":["Citations create false confidence","Model drift after evaluation","Under-resourced clinical review"]}},"retrievedAt":"2026-11-07T22:09:30-06:00","enrichedAt":"2026-11-07T22:21:45-06:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Authors recommend AI-literacy interventions for users; agencies may need resident-facing guidance on verifying AI health answers.","procurementImplications":"Health-information AI procurements should specify grounding, source restriction and rubric-based evaluation, not just citation display.","operatingModelImplications":"Clinical review becomes an ongoing operating function for any AI health-information channel.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.39339","referenceExcerpt":"after correcting for classifier error, an estimated 56.7% of cited statements were supported by their cited page","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}