{"resourceId":"lille-tum-llm-reviewers-published-top-journal-articles-arxiv-2610-31414","versions":[{"version":"external-aa616ba51338c25e2c33bde69ac841e89c5953b9f4bd6be37a43c3a297d09360","resource":{"id":"lille-tum-llm-reviewers-published-top-journal-articles-arxiv-2610-31414","title":"LLMs as reviewers of already-published Nature, Science and Cell papers (arXiv, 22 Oct): 75.6% of 800 reviews demanded major revision or rejection, models disagreed from 17.5% to 100%, and planted flaws were caught 17.0% to 60.5% of the time","organization":"CHU Lille (France) and Technical University of Munich (Le Guellec, Kim, Bentegeac, Hamroun)","sector":"Research integrity, scholarly publishing and peer review","geography":"France and Germany (international journals including U.S. titles; transferable to U.S. research offices and university presses)","publishedAt":"22 October 2026 (arXiv v1, cs.DL)","publicationDate":"2026-10-22","eventDate":null,"sourceName":"arXiv","sourceLabel":"Le Guellec, Kim, Bentegeac and Hamroun, \"Major Revision, Minor Accountability: LLMs as Reviewers of Published Articles in Top-Tier Journals\", arXiv:2610.31414, 22 October 2026","sourceUrl":"https://arxiv.org/abs/2610.31414","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","governance-procurement"],"finding":"The authors assembled 200 CC BY 4.0 articles published in 2026 in 20 general-science, life-science and clinical journals (ten per journal, including Nature, Science and Cell) and asked four current LLMs (GPT-5.6 Sol, GLM-5.3 Flash, Gemini 3.7 Flash and DeepSeek v4 Flash) for blinded peer-review recommendations, without telling them the papers were already published. 605 of 800 reviews (75.6%) recommended major revision or rejection, but per-model rates ranged from 17.5% (Gemini 3.7 Flash) to 100% (GPT-5.6 Sol); the four models agreed unanimously on only 21 of 200 manuscripts, and only on major revision. Recommendations did not differ by journal or prestige. Adding one sentence saying the manuscript was a revision made every model more lenient (pooled mean severity 1.77 to 1.26; Gemini accept rate 15.5% to 68.5%). When one field-specific, severe flaw was planted in each manuscript, detection ranged from 17.0% to 60.5% by model, and inserting the flaw barely changed the recommended decision even when a review identified it. The authors conclude LLM recommendations are a variable, framing- and model-dependent signal that can inform but should not substitute for human editorial judgment, and that any editor using one should monitor it continuously on a local benchmark.","sledRelevance":"Directly relevant to university research offices, faculty senates, grant offices and university presses weighing AI-assisted manuscript or proposal review. It gives concrete evidence that LLM recommendations vary by model and by one sentence of framing, so institutional policies should treat them as advisory signals and require local benchmarking before any use in internal review or pre-submission screening.","evidence":"Read the arXiv abstract page (submitted 22 Oct 2026) and the full PDF: corpus construction (200 articles, 20 journals), model list and selection criteria, blinded prompts, Experiment 1 (decisions: 100%, 90.0%, 17.5%, 95.0% major revision or reject by model), revision-framing experiment (1.77 to 1.26; 49.0% of reviews milder), Experiment 2 (planted-flaw detection 121/200, 88/200, 36/200, 34/200, judged by Claude Opus 5), Experiment 3 (best-review picks), discussion, limitations, AI-use disclosure and funding statement.","architectureImplications":"Any AI-assisted review tool should log model version and prompt framing with each output and support running a local benchmark of known-good and seeded-flaw documents whenever the model changes.","governanceImplications":"Policies should bar AI recommendations from substituting for named human reviewers, require disclosure of AI assistance in internal review, and mandate re-validation after model updates because behaviour is model- and framing-dependent.","securityPrivacyImplications":"The study used open-licensed published papers; sending unpublished manuscripts or proposals to external models raises confidentiality issues the paper does not address.","caveats":"Preprint, not peer reviewed. No expert assessment of review content; the gold standard assumes published papers are sound. Mid-priced rather than largest models were used. Flaw detection was adjudicated by another LLM (Claude Opus 5) rather than humans. Possible training-data contamination for 2026 papers released earlier as preprints. Corpus retrieval, implementation, analysis and drafting were performed substantially by an AI agent under the corresponding author's direction. Results concern biomedical and general-science journals and recommendations, not grant review.","streamIds":["research"],"roles":{"sales":"Lead with governance of AI-assisted review. This 22 October preprint had four LLMs review 200 already-published top-journal papers: 75.6% of reviews demanded major revision or rejection, models ranged from 17.5% to 100%, and planted flaws were caught only 17.0% to 60.5% of the time. Research offices and university presses need defensible rules. Interpretation: a bounded policy and benchmarking engagement fits; this evidence cannot support claims that AI review is reliable or reduces reviewer workload safely.","engineering":"Frame the fit around local benchmarking. The study shows recommendations shift with model choice and with one sentence of framing, so any AI-assisted review workflow needs version and prompt logging and a reusable test set of known-good and seeded-flaw documents. Pre-sales should ask which models are in use, whether confidential manuscripts leave the institution, and who re-tests after model updates. Interpretation: treat converging model concerns as triage signals, not decisions.","delivery":"Treat delivery as building controls around AI-assisted review. Work covers inventorying where AI is used in internal review or pre-submission screening, drafting disclosure and human-accountability rules, building a local seeded-flaw benchmark and scheduling re-tests after model changes. The research office owns policy with research computing running benchmarks. Interpretation: acceptance should track benchmark coverage and documented human sign-off, because the source shows AI recommendations alone are unstable."},"roleDetails":{"sales":{"customerProblem":"Research offices and university presses are under pressure to use AI for manuscript or proposal review but lack evidence on its reliability.","stakeholders":["Vice-president for research","Research integrity officer","University press director","Faculty senate research committee"],"discoveryQuestions":["Where is AI already used in internal or pre-submission review?","What disclosure rules apply to AI-assisted reviews?","How would you detect a change in model behaviour after an update?"],"valueHypothesis":"Clear rules and a local benchmark may let institutions use AI comments as triage while protecting decisions from unstable AI recommendations.","potentialEngagement":"A policy and benchmarking engagement that maps current AI review use and builds a seeded-flaw test set for one review process.","unsupportedClaims":["AI review recommendations are reliable","AI can replace human reviewers","Guaranteed reviewer time savings"]},"engineering":{"fit":"Fits institutions piloting AI-assisted manuscript, proposal or pre-submission review.","architectureIntegration":"Review assistant wrapped with logging of model version and prompt, plus a benchmark harness of known-good and seeded-flaw documents.","prerequisites":["Inventory of models in use","Approved handling of confidential manuscripts","Seeded-flaw test documents"],"constraints":["Preprint evidence only","Biomedical and general-science corpus","Behaviour changes with model versions"],"security":"Keep unpublished manuscripts and proposals on approved models or infrastructure and log all AI processing.","proofOfValue":["Measure specificity on known-good documents","Measure detection of seeded flaws","Re-run benchmark after each model update"]},"delivery":{"work":"Inventory AI use, draft rules, build benchmark, run baseline tests and train reviewers on interpreting AI comments.","dependencies":["Faculty governance approval","Domain experts to seed flaws","Access to the models in use"],"ownership":"Office of research or research integrity, with research computing operating benchmarks.","skillsAdoption":"Reviewers learn to use AI comments as prompts and record human decisions; staff learn to maintain benchmarks.","governanceCheckpoints":["Policy approval","Benchmark results reviewed before any pilot","Re-validation after model change"],"acceptanceCriteria":["Proposed: every AI-assisted review has a named human decision-maker","Proposed: benchmark re-run within an agreed period after model updates","Proposed: disclosure recorded for all AI-assisted reviews"],"risks":["Over-reliance on harsh or lenient AI verdicts","Confidential data exposure","Silent behaviour change after updates"]}},"retrievedAt":"2026-10-25T22:05:16-05:00","enrichedAt":"2026-10-25T22:05:17-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"AI does not remove the need for expert reviewer capacity; staff and faculty need training to interpret AI comments as prompts, not verdicts.","procurementImplications":"When licensing AI review or manuscript-screening tools, require vendor evidence of specificity and sensitivity on seeded-flaw benchmarks and the right to re-test after model updates.","operatingModelImplications":"Use AI comments, especially concerns that converge across models, to direct human attention; keep decisions with accountable human reviewers or editors.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.31414","referenceExcerpt":"605 of 800 reviews (75.6%) recommended major revision or rejection, but the rate at which any single model did so ranged from 17.5% to 100%","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}