From the Research Universities edition of October 25, 2026
LLMs as reviewers of already-published Nature, Science and Cell papers (arXiv, 22 Oct): 75.6% of 800 reviews demanded major revision or rejection, models disagreed from 17.5% to 100%, and planted flaws were caught 17.0% to 60.5% of the time
CHU Lille (France) and Technical University of Munich (Le Guellec, Kim, Bentegeac, Hamroun) · Research integrity, scholarly publishing and peer review · France and Germany (international journals including U.S. titles; transferable to U.S. research offices and university presses)
- Publisher
- arXiv
- Original publication
- 22 October 2026 (arXiv v1, cs.DL)
- Source retrieved
- 2026-10-26
What happened
The authors assembled 200 CC BY 4.0 articles published in 2026 in 20 general-science, life-science and clinical journals (ten per journal, including Nature, Science and Cell) and asked four current LLMs (GPT-5.6 Sol, GLM-5.3 Flash, Gemini 3.7 Flash and DeepSeek v4 Flash) for blinded peer-review recommendations, without telling them the papers were already published. 605 of 800 reviews (75.6%) recommended major revision or rejection, but per-model rates ranged from 17.5% (Gemini 3.7 Flash) to 100% (GPT-5.6 Sol); the four models agreed unanimously on only 21 of 200 manuscripts, and only on major revision. Recommendations did not differ by journal or prestige. Adding one sentence saying the manuscript was a revision made every model more lenient (pooled mean severity 1.77 to 1.26; Gemini accept rate 15.5% to 68.5%). When one field-specific, severe flaw was planted in each manuscript, detection ranged from 17.0% to 60.5% by model, and inserting the flaw barely changed the recommended decision even when a review identified it. The authors conclude LLM recommendations are a variable, framing- and model-dependent signal that can inform but should not substitute for human editorial judgment, and that any editor using one should monitor it continuously on a local benchmark.
Why it matters
Directly relevant to university research offices, faculty senates, grant offices and university presses weighing AI-assisted manuscript or proposal review. It gives concrete evidence that LLM recommendations vary by model and by one sentence of framing, so institutional policies should treat them as advisory signals and require local benchmarking before any use in internal review or pre-submission screening.
Evidence and measured results
Read the arXiv abstract page (submitted 22 Oct 2026) and the full PDF: corpus construction (200 articles, 20 journals), model list and selection criteria, blinded prompts, Experiment 1 (decisions: 100%, 90.0%, 17.5%, 95.0% major revision or reject by model), revision-framing experiment (1.77 to 1.26; 49.0% of reviews milder), Experiment 2 (planted-flaw detection 121/200, 88/200, 36/200, 34/200, judged by Claude Opus 5), Experiment 3 (best-review picks), discussion, limitations, AI-use disclosure and funding statement.
Limitations and uncertainty
Preprint, not peer reviewed. No expert assessment of review content; the gold standard assumes published papers are sound. Mid-priced rather than largest models were used. Flaw detection was adjudicated by another LLM (Claude Opus 5) rather than humans. Possible training-data contamination for 2026 papers released earlier as preprints. Corpus retrieval, implementation, analysis and drafting were performed substantially by an AI agent under the corresponding author's direction. Results concern biomedical and general-science journals and recommendations, not grant review.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-25; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Research offices and university presses are under pressure to use AI for manuscript or proposal review but lack evidence on its reliability.
- Stakeholders
- Vice-president for research
- Research integrity officer
- University press director
- Faculty senate research committee
- Discovery questions
- Where is AI already used in internal or pre-submission review?
- What disclosure rules apply to AI-assisted reviews?
- How would you detect a change in model behaviour after an update?
- Value hypothesis
- Clear rules and a local benchmark may let institutions use AI comments as triage while protecting decisions from unstable AI recommendations.
- Potential engagement
- A policy and benchmarking engagement that maps current AI review use and builds a seeded-flaw test set for one review process.
- Unsupported claims
- AI review recommendations are reliable
- AI can replace human reviewers
- Guaranteed reviewer time savings
Pre-sales engineering
Role takeaway
- Fit
- Fits institutions piloting AI-assisted manuscript, proposal or pre-submission review.
- Architecture and integration
- Review assistant wrapped with logging of model version and prompt, plus a benchmark harness of known-good and seeded-flaw documents.
- Prerequisites
- Inventory of models in use
- Approved handling of confidential manuscripts
- Seeded-flaw test documents
- Constraints
- Preprint evidence only
- Biomedical and general-science corpus
- Behaviour changes with model versions
- Security
- Keep unpublished manuscripts and proposals on approved models or infrastructure and log all AI processing.
- Proof of value
- Measure specificity on known-good documents
- Measure detection of seeded flaws
- Re-run benchmark after each model update
Delivery
Role takeaway
- Work
- Inventory AI use, draft rules, build benchmark, run baseline tests and train reviewers on interpreting AI comments.
- Dependencies
- Faculty governance approval
- Domain experts to seed flaws
- Access to the models in use
- Ownership
- Office of research or research integrity, with research computing operating benchmarks.
- Skills and adoption
- Reviewers learn to use AI comments as prompts and record human decisions; staff learn to maintain benchmarks.
- Governance checkpoints
- Policy approval
- Benchmark results reviewed before any pilot
- Re-validation after model change
- Acceptance criteria
- Proposed: every AI-assisted review has a named human decision-maker
- Proposed: benchmark re-run within an agreed period after model updates
- Proposed: disclosure recorded for all AI-assisted reviews
- Risks
- Over-reliance on harsh or lenient AI verdicts
- Confidential data exposure
- Silent behaviour change after updates
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Any AI-assisted review tool should log model version and prompt framing with each output and support running a local benchmark of known-good and seeded-flaw documents whenever the model changes.
Governance
Who approves, reviews and stays accountable for outcomes?
Policies should bar AI recommendations from substituting for named human reviewers, require disclosure of AI assistance in internal review, and mandate re-validation after model updates because behaviour is model- and framing-dependent.
Security and privacy
What data, permissions and controls need testing?
The study used open-licensed published papers; sending unpublished manuscripts or proposals to external models raises confidentiality issues the paper does not address.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
AI does not remove the need for expert reviewer capacity; staff and faculty need training to interpret AI comments as prompts, not verdicts.
Procurement
What should contracts, pricing and exit terms secure?
When licensing AI review or manuscript-screening tools, require vendor evidence of specificity and sensitivity on seeded-flaw benchmarks and the right to re-test after model updates.
Operating model
Which teams own the service once it runs?
Use AI comments, especially concerns that converge across models, to direct human attention; keep decisions with accountable human reviewers or editors.
Publication history
- 2026-10-25Research Universities · Issue 474 resources
Stable resource ID: lille-tum-llm-reviewers-published-top-journal-articles-arxiv-2610-31414