From the Research Universities edition of October 18, 2026
Eye study (Adelaide): LLMs caught 81.7% of planted manuscript inconsistencies and flagged mostly genuine errors, but missed some systematically
Adelaide University and Royal Adelaide Hospital (La, Paris, Agarwal, Chan, Selva) · Academic publishing and research integrity (ophthalmology) · Australia; transferable to U.S. journal offices and pre-submission review with domain limits
- Publisher
- Eye (Nature Portfolio)
- Original publication
- 9 October 2026
- Source retrieved
- 2026-10-19
What happened
Researchers planted one inconsistency (percentage/count, sample size/subgrouping or statistical reporting) in 30 of 40 published ophthalmic articles and had four LLMs review all 40 with one prompt. Across 120 assessments of manipulated manuscripts, 98 planted errors were detected (81.7%), ranging from 96.7% (GPT-5.6) to 63.3% (GPT-5.5). Once detected, all were localised and 97 of 98 correctly explained and corrected. Of 355 additional flags, 337 (94.9%) were genuine pre-existing inconsistencies, 158 unique after de-duplication, mostly rounding discrepancies for the top model. The authors conclude that missing errors, not interpreting them, is the main limit and position LLMs as complementary pre-submission or editorial screening.
Why it matters
Research offices, journals hosted by universities and pre-submission support services are considering AI checks. This is measured evidence for a narrow support task of the kind the new EU guidance allows, not for judging merit.
Evidence and measured results
Opened the full open-access brief communication on nature.com: methods, Tables 1 and 2 as described in the text, and conclusions.
Limitations and uncertainty
Small sample (40 articles, 30 planted errors, one discipline); planted errors were of three simple types; model versions change quickly; datasets available only on request.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-18; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Editors and research-support offices cannot check every number and subgroup in manuscripts before submission or review.
- Stakeholders
- Editors of university-hosted journals
- Research support or research development office
- Research integrity officer
- Library publishing services
- Discovery questions
- Which inconsistency types cause the most corrections or retractions in your field?
- Who would adjudicate AI flags?
- Are unpublished manuscripts allowed in the AI tools you are considering?
- Value hypothesis
- A screening step may catch some numeric and reporting inconsistencies before submission, reducing routine checking effort.
- Potential engagement
- A pilot that runs a planted-error test in one discipline and sets a screening workflow with human adjudication.
- Unsupported claims
- AI can replace peer review
- Detection rates transfer to other fields or error types
- Reduced retractions or corrections
Pre-sales engineering
Role takeaway
- Fit
- Fits pre-submission and editorial screening of quantitative manuscripts; limited to the error types tested.
- Architecture and integration
- Add an LLM consistency check to submission or pre-submission workflows, with flags routed to human reviewers.
- Prerequisites
- Approved AI tool for manuscript text
- Planted-error test set in the target field
- Flag adjudication process
- Constraints
- Tested only on published ophthalmology articles
- Three planted error types
- Unpublished manuscripts need a confidentiality-safe tool
- Security
- Not tested for unpublished work; use a governed tool that does not retain or train on manuscripts.
- Proof of value
- Measure detection on a local planted-error set
- Track the share of extra flags that are genuine
- Compare two models on the same set
Delivery
Role takeaway
- Work
- Configure the checking tool, define covered error types and set up a flag-review queue.
- Dependencies
- Tool approval by IT and integrity office
- Editor or staff time to adjudicate
- Local test set
- Ownership
- Journal editor or research-support lead.
- Skills and adoption
- Staff learn to adjudicate AI flags and recognise error types the tool misses.
- Governance checkpoints
- Tool approval for manuscript data
- Periodic re-testing after model updates
- Disclosure to authors of AI screening
- Acceptance criteria
- Proposal: local planted-error sensitivity meets a threshold set before deployment
- Proposal: share of genuine flags tracked monthly
- Proposal: every flag adjudicated by a person
- Risks
- False assurance from missed errors
- Confidentiality breaches with unpublished work
- Model updates changing performance
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Single-prompt review by multiple models gave different sensitivity; running more than one model or a structured checklist may matter more than any one model.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports AI as a quality-control layer with human adjudication of flags; does not support AI accept or reject decisions.
Security and privacy
What data, permissions and controls need testing?
Used published articles; unpublished manuscripts would raise confidentiality questions not tested here.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Could reduce routine checking burden for editors and authors, though staff still adjudicate flags; not measured.
Procurement
What should contracts, pricing and exit terms secure?
Model choice changed sensitivity from 63.3% to 96.7%; buyers should test candidate tools on planted-error sets from their own field.
Operating model
Which teams own the service once it runs?
Fits as a pre-submission or editorial screening step with human adjudication, not a replacement for review.
Publication history
- 2026-10-18Research Universities · Issue 404 resources
Stable resource ID: adelaide-llms-manuscript-consistency-checkers-eye-20261009