From the Google edition of November 7, 2026
Northwestern preprint: Google AI Overviews (Gemini) on 3,611 health questions cite fewer government/academic sources, ~57% of cited statements supported, 22% include anticipated harmful claims
Northwestern University (Agnoli, Reilly, Barna, Horvát) · Public health information; health & human services communications · United States (U.S. English locale searches)
- Publisher
- arXiv
- Original publication
- 2026-10-29 (arXiv v1)
- Source retrieved
- 2026-11-08
What happened
Source finding (Preprint, not peer reviewed): researchers queried 3,611 HealthBench health questions via SerpAPI (Sept. 16–18, 2026). AI Overviews appeared for 96.4% of questions; only 44.4% of the traditional top-five results were cited by the AI Overview, which over-represented commercial publishers versus academic and government sources (e.g., NHS share 1.5% vs 3.1%). After correcting for classifier error, an estimated 56.7% of cited statements were supported by their cited page; 20.8% of statements were uncited factual claims. Against physician rubrics, AI Overviews earned 38.2% of points (accuracy 54.2%, completeness 21.6%), and 22.2% of those with an anticipated harmful claim contained it. Accuracy was unrelated to grounding.
Why it matters
Public health departments and HHS agencies depend on residents finding authoritative guidance; this study indicates Gemini-powered AI Overviews may displace government health sources in search answers. Why now: it complements the Nov. 5 Princeton voting-information audit in this edition, giving a second, larger independent measurement of Google's AI answers on public-interest information, and it is not yet in the archive. Cross-tagged to health-human-services because the measured domain is health information.
Evidence and measured results
Preprint with detailed methods: 3,611 questions, 3,474 usable overviews, 24,060 statements; JAMA and HONcode source-quality coding (automated, kappa .68 vs humans); HallOumi grounding classifier (kappa .61, corrected with Rogan–Gladen); GPT-4.1 grading against HealthBench physician rubrics (macro-F1 .707 vs physicians). Data and code on GitHub/Zenodo. Baseline is Google's own top-five traditional results.
Limitations and uncertainty
Preprint, not peer reviewed. HealthBench questions are longer than typical searches; automated coders and an LLM grader were used; three-day scrape; AI Overviews change frequently. Measures public Google Search, not an agency-configured Gemini deployment, so transferability to grounded enterprise builds is limited.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-07; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Residents receive AI-generated health answers that may omit or contradict agency guidance and under-cite government sources.
- Stakeholders
- Public health communications directors
- Medical directors/health officers
- HHS agency CIOs
- Medicaid member-services leads
- Discovery questions
- Do you know what AI search tells residents about your top health topics?
- Would a source-restricted assistant be acceptable to your medical director?
- What clinical review capacity exists for AI outputs?
- Is your guidance published in structured, citable form?
- Value hypothesis
- Interpretation: a bounded evaluation could quantify how often AI answers misstate agency guidance on priority topics; the source does not show any remediation effect.
- Potential engagement
- Bounded: adapt the published open pipeline to 100–200 agency priority questions and report grounding and rubric scores.
- Unsupported claims
- That agency-deployed Gemini assistants share these error rates
- That Google will change AI Overviews sourcing
- Any health-outcome impact
Pre-sales engineering
Role takeaway
- Fit
- Fit as an evaluation method and risk input for any Gemini/Vertex AI health-information use case.
- Architecture and integration
- Retrieval restricted to approved agency/federal health content; claim-level verification against retrieved passages; rubric-based grading with clinician sampling.
- Prerequisites
- Curated corpus of approved health guidance
- Clinician-written rubrics for priority questions
- Logging of answers and citations for review
- Constraints
- Public AI Overviews are not configurable by agencies
- Study used English, U.S. locale only
- LLM grader and classifiers have known error rates
- Security
- No PHI needed for evaluation; production assistants touching member data need HIPAA-aligned controls beyond this study.
- Proof of value
- Measure supported-statement rate on agency questions
- Test whether uncited claims appear
- Compare with a source-restricted configuration
Delivery
Role takeaway
- Work
- Run a grounding and clinical-accuracy evaluation before any health-information assistant goes live.
- Dependencies
- Medical director sign-off
- Content owners for guidance pages
- Evaluation tooling (open pipeline)
- Ownership
- Public health communications lead with the health officer's clinical review.
- Skills and adoption
- Staff need training to review AI outputs against rubrics and recognize omission errors, which dominated lost points in the study.
- Governance checkpoints
- Approve source corpus
- Clinical review of sample outputs
- Go/no-go on grounding threshold
- Acceptance criteria
- Proposed: supported-statement rate above an agency-set threshold on priority questions
- Proposed: zero anticipated-harmful claims in a reviewed sample
- Proposed: emergency-referral questions route to care in all reviewed cases
- Risks
- Citations create false confidence
- Model drift after evaluation
- Under-resourced clinical review
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
For agency-built Gemini/Vertex AI health assistants, the finding that citations did not predict accuracy argues for grounding restricted to agency-approved sources plus independent claim verification, rather than trusting the presence of citations.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports agency policies that warn residents to verify AI-generated health answers and that require source-restriction and human review for any agency-deployed health assistant. Authors recommend Google prefer academic/government domains and suppress AI Overviews for complex or risky questions; those are recommendations, not Google commitments.
Security and privacy
What data, permissions and controls need testing?
No personal data in the study; the risk is harmful or ungrounded health guidance reaching residents.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Authors recommend AI-literacy interventions for users; agencies may need resident-facing guidance on verifying AI health answers.
Procurement
What should contracts, pricing and exit terms secure?
Health-information AI procurements should specify grounding, source restriction and rubric-based evaluation, not just citation display.
Operating model
Which teams own the service once it runs?
Clinical review becomes an ongoing operating function for any AI health-information channel.
Publication history
- 2026-11-07Google · Issue 106 resources
Stable resource ID: arxiv-northwestern-google-ai-overviews-health-grounding-audit-preprint-20261029