From the Anthropic edition of November 7, 2026
Manchester-led preprint: Claude Code and other AI tools miss literature without library access and over-classify passing mentions; Claude's quotations were accurate
University of Manchester and University of Washington researchers (Al Kalbani, Schultz, Anderson-Frey, Zheng, McGuire) · Higher education / research enterprise · United Kingdom and United States
- Publisher
- arXiv
- Original publication
- 2026-11-03 (arXiv v1)
- Source retrieved
- 2026-11-08
What happened
Source finding (preprint, not peer reviewed): against a 102-article benchmark review, five tools searched for literature; Claude's web interface (Claude Opus 5) found 7 and Claude Code 15 without library login, while Codex found 42. With a University of Manchester library login, Claude Code found 79 and Codex 82. Given all 102 PDFs, Claude Code, Codex and NotebookLM all identified the 18 substantial articles, but misclassified 40% (Claude), 77% and 79% of the 84 passing mentions; Claude's Cohen's kappa was 0.34. All 101 Claude quotations appeared word for word (3 page numbers wrong). Blind expert ratings of syntheses diverged: Reviewer A rated Claude highest (4.32 mean) while Reviewer B rated the three tools similarly (2.78-2.92). Authors conclude tools can assist but did not synthesize the literature effectively. Why now: posted Nov 3; not previously archived.
Why it matters
Relevant to research universities and libraries deciding how faculty and graduate students use Claude Code or Claude research modes for literature reviews: access through institutional subscriptions changed results more than tool choice, and expert verification remained necessary. Also informs library licensing and bot-access policies.
Evidence and measured results
One topic (Spanish plume meteorology), one run per tool per task, two reviewers who disagreed (exact agreement 24%), Jul-Aug 2026 tool versions, entry-level paid plans, login state not uniformly controlled in the first search comparison. Data and prompts are public on Zenodo and GitHub per the authors.
Limitations and uncertainty
Preprint, not peer reviewed; multi-vendor study, and only Claude results are claimed for this stream; small and single-domain; results may not transfer to other fields or newer model versions.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-08; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Faculty and students use AI tools for literature reviews without evidence about coverage gaps or interpretation errors.
- Stakeholders
- Vice president for research
- University librarian
- Research integrity officer
- Graduate school dean
- Discovery questions
- Do researchers use Claude Code or research modes for literature reviews today?
- How do agents access licensed library content?
- Is there guidance on verifying AI-generated citations?
- Value hypothesis
- Guided use with library access and verification steps can make Claude useful for search and extraction while limiting integrity risk.
- Potential engagement
- Benchmark pilot with two or three labs measuring Claude coverage and verification effort on a known corpus, producing campus guidance.
- Unsupported claims
- Time savings (not measured)
- That Claude can write literature reviews unaided
- Generalization beyond one meteorology topic
Pre-sales engineering
Role takeaway
- Fit
- Fits campuses with Claude for Education or Claude Code licenses and active research libraries.
- Architecture and integration
- Connect agent workflows to library proxy/authentication within license terms, export structured extraction tables and run automated quotation checks.
- Prerequisites
- Library licensing review
- Researcher accounts on an institutional Claude tenant
- Benchmark corpus for evaluation
- Constraints
- Publisher terms may prohibit automated access
- Bot blocking varies by publisher
- Results tied to specific tool versions
- Security
- Avoid sharing personal library credentials with agents outside approved configurations; log agent access.
- Proof of value
- Measure recall against a known review with and without library access
- Run automated quotation verification on extraction output
- Compare passing-mention classification against expert labels
Delivery
Role takeaway
- Work
- Draft and pilot campus guidance for AI-assisted literature reviews based on the study's recommendations.
- Dependencies
- Library licensing staff
- Research integrity office
- Participating labs
- Ownership
- University library with the research integrity office.
- Skills and adoption
- Workshops for graduate students on verification, prompt detail and auditing search coverage.
- Governance checkpoints
- License compliance review before agent library access
- Integrity office approval of guidance
- Annual re-test with current tool versions
- Acceptance criteria
- Proposed: guidance published and adopted by pilot labs
- Proposed: pilot reports recall and verification time on benchmark corpus
- Risks
- License violations through automated access
- Overreliance on confident syntheses
- Rapid tool changes outdate guidance
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Agents running inside authenticated institutional sessions reach materially more literature; campuses need to decide how agents use library credentials and respect publisher terms.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports research-integrity guidance requiring human verification of AI-generated literature claims and audit of search coverage.
Security and privacy
What data, permissions and controls need testing?
Letting coding agents act through a researcher's browser session and library login raises credential-use and license-compliance questions.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Authors warn outsourcing reviews undermines PhD students' learning; training must keep core research skills.
Procurement
What should contracts, pricing and exit terms secure?
Library subscriptions materially affected agent results; institutions may weigh license terms for AI agent access in renewals.
Operating model
Which teams own the service once it runs?
Libraries gain a role in configuring and auditing AI research workflows.
Publication history
- 2026-11-07Anthropic · Issue 106 resources
Stable resource ID: manchester-llm-literature-review-102-article-benchmark-claude-20261103