{"resourceId":"adelaide-llms-manuscript-consistency-checkers-eye-20261009","versions":[{"version":"external-0428d74b4c5bc87a05a1c468436ccf5bb195c0174e48feaf43cfabd61f5bde10","resource":{"id":"adelaide-llms-manuscript-consistency-checkers-eye-20261009","title":"Eye study (Adelaide): LLMs caught 81.7% of planted manuscript inconsistencies and flagged mostly genuine errors, but missed some systematically","organization":"Adelaide University and Royal Adelaide Hospital (La, Paris, Agarwal, Chan, Selva)","sector":"Academic publishing and research integrity (ophthalmology)","geography":"Australia; transferable to U.S. journal offices and pre-submission review with domain limits","publishedAt":"9 October 2026","publicationDate":"2026-10-09","eventDate":null,"sourceName":"Eye (Nature Portfolio)","sourceLabel":"Brief communication, open access","sourceUrl":"https://www.nature.com/articles/s41433-026-04965-5","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","governance-procurement"],"finding":"Researchers planted one inconsistency (percentage/count, sample size/subgrouping or statistical reporting) in 30 of 40 published ophthalmic articles and had four LLMs review all 40 with one prompt. Across 120 assessments of manipulated manuscripts, 98 planted errors were detected (81.7%), ranging from 96.7% (GPT-5.6) to 63.3% (GPT-5.5). Once detected, all were localised and 97 of 98 correctly explained and corrected. Of 355 additional flags, 337 (94.9%) were genuine pre-existing inconsistencies, 158 unique after de-duplication, mostly rounding discrepancies for the top model. The authors conclude that missing errors, not interpreting them, is the main limit and position LLMs as complementary pre-submission or editorial screening.","sledRelevance":"Research offices, journals hosted by universities and pre-submission support services are considering AI checks. This is measured evidence for a narrow support task of the kind the new EU guidance allows, not for judging merit.","evidence":"Opened the full open-access brief communication on nature.com: methods, Tables 1 and 2 as described in the text, and conclusions.","architectureImplications":"Single-prompt review by multiple models gave different sensitivity; running more than one model or a structured checklist may matter more than any one model.","governanceImplications":"Supports AI as a quality-control layer with human adjudication of flags; does not support AI accept or reject decisions.","securityPrivacyImplications":"Used published articles; unpublished manuscripts would raise confidentiality questions not tested here.","caveats":"Small sample (40 articles, 30 planted errors, one discipline); planted errors were of three simple types; model versions change quickly; datasets available only on request.","streamIds":["research"],"roles":{"sales":"Lead with a narrow, measurable check. Four LLMs caught 81.7% of planted inconsistencies in published ophthalmology papers and most other flags were real errors, but sensitivity ranged from 63.3% to 96.7% by model. Research offices and university-hosted journals can offer this as pre-submission quality control. Interpretation: one small study in one field supports a pilot with planted-error testing, not claims that AI replaces peer review.","engineering":"Treat this as a test protocol for any manuscript-checking tool. Plant known errors of set types, run the tool, and record detection, localisation and correction separately, along with how many extra flags are genuine. Pre-sales should insist on a field-specific planted-error test because model choice changed sensitivity sharply. Interpretation; the study used published papers, so confidentiality handling for unpublished work still needs design.","delivery":"Frame delivery as a screening step with human adjudication. Work would set up the checking tool, define which error types it covers, and route flags to an editor or research-support staff member. The journal editor or research-support lead owns the step. Interpretation: acceptance should be based on local planted-error sensitivity and the share of genuine flags, since the source covers one field and three error types."},"roleDetails":{"sales":{"customerProblem":"Editors and research-support offices cannot check every number and subgroup in manuscripts before submission or review.","stakeholders":["Editors of university-hosted journals","Research support or research development office","Research integrity officer","Library publishing services"],"discoveryQuestions":["Which inconsistency types cause the most corrections or retractions in your field?","Who would adjudicate AI flags?","Are unpublished manuscripts allowed in the AI tools you are considering?"],"valueHypothesis":"A screening step may catch some numeric and reporting inconsistencies before submission, reducing routine checking effort.","potentialEngagement":"A pilot that runs a planted-error test in one discipline and sets a screening workflow with human adjudication.","unsupportedClaims":["AI can replace peer review","Detection rates transfer to other fields or error types","Reduced retractions or corrections"]},"engineering":{"fit":"Fits pre-submission and editorial screening of quantitative manuscripts; limited to the error types tested.","architectureIntegration":"Add an LLM consistency check to submission or pre-submission workflows, with flags routed to human reviewers.","prerequisites":["Approved AI tool for manuscript text","Planted-error test set in the target field","Flag adjudication process"],"constraints":["Tested only on published ophthalmology articles","Three planted error types","Unpublished manuscripts need a confidentiality-safe tool"],"security":"Not tested for unpublished work; use a governed tool that does not retain or train on manuscripts.","proofOfValue":["Measure detection on a local planted-error set","Track the share of extra flags that are genuine","Compare two models on the same set"]},"delivery":{"work":"Configure the checking tool, define covered error types and set up a flag-review queue.","dependencies":["Tool approval by IT and integrity office","Editor or staff time to adjudicate","Local test set"],"ownership":"Journal editor or research-support lead.","skillsAdoption":"Staff learn to adjudicate AI flags and recognise error types the tool misses.","governanceCheckpoints":["Tool approval for manuscript data","Periodic re-testing after model updates","Disclosure to authors of AI screening"],"acceptanceCriteria":["Proposal: local planted-error sensitivity meets a threshold set before deployment","Proposal: share of genuine flags tracked monthly","Proposal: every flag adjudicated by a person"],"risks":["False assurance from missed errors","Confidentiality breaches with unpublished work","Model updates changing performance"]}},"retrievedAt":"2026-10-18T21:59:03-05:00","enrichedAt":"2026-10-18T21:59:03-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Could reduce routine checking burden for editors and authors, though staff still adjudicate flags; not measured.","procurementImplications":"Model choice changed sensitivity from 63.3% to 96.7%; buyers should test candidate tools on planted-error sets from their own field.","operatingModelImplications":"Fits as a pre-submission or editorial screening step with human adjudication, not a replacement for review.","sourceVerification":{"openedUrl":"https://www.nature.com/articles/s41433-026-04965-5","referenceExcerpt":"failure of systematic detection remains the principal limitation across current LLMs","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}