{"resourceId":"manchester-llm-literature-review-102-article-benchmark-claude-20261103","versions":[{"version":"external-19a2f2f033744f69bbd6a0d7a18fdd19b6f9ae7939bcb8aba5c07dcc147acda3","resource":{"id":"manchester-llm-literature-review-102-article-benchmark-claude-20261103","title":"Manchester-led preprint: Claude Code and other AI tools miss literature without library access and over-classify passing mentions; Claude's quotations were accurate","organization":"University of Manchester and University of Washington researchers (Al Kalbani, Schultz, Anderson-Frey, Zheng, McGuire)","sector":"Higher education / research enterprise","geography":"United Kingdom and United States","publishedAt":"2026-11-03 (arXiv v1)","publicationDate":"2026-11-03","eventDate":null,"sourceName":"arXiv","sourceLabel":"Preprint, not peer reviewed","sourceUrl":"https://arxiv.org/abs/2611.05459","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","developers-agents"],"finding":"Source finding (preprint, not peer reviewed): against a 102-article benchmark review, five tools searched for literature; Claude's web interface (Claude Opus 5) found 7 and Claude Code 15 without library login, while Codex found 42. With a University of Manchester library login, Claude Code found 79 and Codex 82. Given all 102 PDFs, Claude Code, Codex and NotebookLM all identified the 18 substantial articles, but misclassified 40% (Claude), 77% and 79% of the 84 passing mentions; Claude's Cohen's kappa was 0.34. All 101 Claude quotations appeared word for word (3 page numbers wrong). Blind expert ratings of syntheses diverged: Reviewer A rated Claude highest (4.32 mean) while Reviewer B rated the three tools similarly (2.78-2.92). Authors conclude tools can assist but did not synthesize the literature effectively. Why now: posted Nov 3; not previously archived.","sledRelevance":"Relevant to research universities and libraries deciding how faculty and graduate students use Claude Code or Claude research modes for literature reviews: access through institutional subscriptions changed results more than tool choice, and expert verification remained necessary. Also informs library licensing and bot-access policies.","evidence":"One topic (Spanish plume meteorology), one run per tool per task, two reviewers who disagreed (exact agreement 24%), Jul-Aug 2026 tool versions, entry-level paid plans, login state not uniformly controlled in the first search comparison. Data and prompts are public on Zenodo and GitHub per the authors.","architectureImplications":"Agents running inside authenticated institutional sessions reach materially more literature; campuses need to decide how agents use library credentials and respect publisher terms.","governanceImplications":"Supports research-integrity guidance requiring human verification of AI-generated literature claims and audit of search coverage.","securityPrivacyImplications":"Letting coding agents act through a researcher's browser session and library login raises credential-use and license-compliance questions.","caveats":"Preprint, not peer reviewed; multi-vendor study, and only Claude results are claimed for this stream; small and single-domain; results may not transfer to other fields or newer model versions.","streamIds":["anthropic","research"],"roles":{"sales":"Sales framing: offer research-workflow guidance, not productivity guarantees. Research offices and libraries adopting Claude for Education or Claude Code can use this study to set realistic expectations: Claude quoted accurately and classified better than peers here, yet still misjudged 40% of passing mentions and needed library access to find most papers. A bounded pilot measuring coverage and verification time on a known corpus is a credible engagement. Cross-tagged to Research Universities for its research-enterprise relevance.","engineering":"Engineering framing: design for access and verification. Evaluate how Claude Code or research modes authenticate to library resources, whether that use complies with publisher licenses, and how outputs carry checkable citations. Ask for structured extraction tables, because the authors found databases easier to verify automatically than prose syntheses. Build a quotation-check script into the workflow and log which sources the agent could not reach, as the study recommends auditing search records.","delivery":"Delivery framing: package the authors' eight recommendations into campus guidance. Library and research-integrity offices should own a short standard covering subscription-backed search, auditing blocked sources, treating syntheses as claim lists, expert review and prompt detail. Pilot with a few labs using a known review as a benchmark, measure coverage and correction effort, and update guidance as tool versions change, since results reflect mid-2026 versions."},"roleDetails":{"sales":{"customerProblem":"Faculty and students use AI tools for literature reviews without evidence about coverage gaps or interpretation errors.","stakeholders":["Vice president for research","University librarian","Research integrity officer","Graduate school dean"],"discoveryQuestions":["Do researchers use Claude Code or research modes for literature reviews today?","How do agents access licensed library content?","Is there guidance on verifying AI-generated citations?"],"valueHypothesis":"Guided use with library access and verification steps can make Claude useful for search and extraction while limiting integrity risk.","potentialEngagement":"Benchmark pilot with two or three labs measuring Claude coverage and verification effort on a known corpus, producing campus guidance.","unsupportedClaims":["Time savings (not measured)","That Claude can write literature reviews unaided","Generalization beyond one meteorology topic"]},"engineering":{"fit":"Fits campuses with Claude for Education or Claude Code licenses and active research libraries.","architectureIntegration":"Connect agent workflows to library proxy/authentication within license terms, export structured extraction tables and run automated quotation checks.","prerequisites":["Library licensing review","Researcher accounts on an institutional Claude tenant","Benchmark corpus for evaluation"],"constraints":["Publisher terms may prohibit automated access","Bot blocking varies by publisher","Results tied to specific tool versions"],"security":"Avoid sharing personal library credentials with agents outside approved configurations; log agent access.","proofOfValue":["Measure recall against a known review with and without library access","Run automated quotation verification on extraction output","Compare passing-mention classification against expert labels"]},"delivery":{"work":"Draft and pilot campus guidance for AI-assisted literature reviews based on the study's recommendations.","dependencies":["Library licensing staff","Research integrity office","Participating labs"],"ownership":"University library with the research integrity office.","skillsAdoption":"Workshops for graduate students on verification, prompt detail and auditing search coverage.","governanceCheckpoints":["License compliance review before agent library access","Integrity office approval of guidance","Annual re-test with current tool versions"],"acceptanceCriteria":["Proposed: guidance published and adopted by pilot labs","Proposed: pilot reports recall and verification time on benchmark corpus"],"risks":["License violations through automated access","Overreliance on confident syntheses","Rapid tool changes outdate guidance"]}},"retrievedAt":"2026-11-08T04:04:30Z","enrichedAt":"2026-11-08T04:08:54Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Authors warn outsourcing reviews undermines PhD students' learning; training must keep core research skills.","procurementImplications":"Library subscriptions materially affected agent results; institutions may weigh license terms for AI agent access in renewals.","operatingModelImplications":"Libraries gain a role in configuring and auditing AI research workflows.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2611.05459","referenceExcerpt":"incorrectly classified 40% to 79% of the 84 passing mentions as substantial","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}