From the Research Universities edition of October 29, 2026
Preprint: 15 LLMs found 51% to 99% of errors inserted into 2026 clinical trial reports, versus 27% for untrained human reviewers in an earlier study, but also flagged problems in unaltered controls
ResearchHub Foundation, University of Southern California, Western University and OrganoBit BioIntelligence · Scholarly peer review and research integrity (clinical research) · United States, Canada and Singapore (authors); international journals
- Publisher
- medRxiv
- Original publication
- 28 October 2026 (medRxiv posting)
- Source retrieved
- 2026-10-30
What happened
The authors repeated the largest inserted-error peer review experiment, in which 607 untrained human reviewers found a mean of 27% of major errors in three trial reports, with 15 LLMs from six developers, including four open-weight models. On the original papers, LLM major-error detection scores ranged from 53% to 96%. To reduce memorization, they inserted the same 14 error types into three randomized trials published in 2026: scores ranged from 51% to 99% (66% to 87% for open-weight models), with Opus 5.5 highest at 99%. On unmodified control versions, models flagged 2% to 29% of error types (overall specificity 83.0%), and models that found more errors also flagged more on controls (Spearman rho -0.77). Blinded model judges scored reviews with 94.4% and 97.9% agreement. Review times were 1.4 to 16.3 minutes and estimated costs $0.005 to $2.24 per review for the 12 models timed. The authors recommend AI-assisted methodological checks as a complement to human review.
Why it matters
Research integrity offices, clinical research units and university presses considering AI pre-submission checks get a controlled benchmark that includes false-positive rates on clean manuscripts. It contrasts with recent cautionary studies of AI review in this stream, showing narrow, checkable tasks fare better than overall judgments.
Evidence and measured results
Read the abstract, results, discussion, limitations and competing interests in the full-text PDF. Supplementary tables were not opened. The human comparison is historical.
Limitations and uncertainty
Preprint, not peer reviewed. Small set of six trial reports and predefined, in-text error types; human comparison uses historical reviewers on different manuscripts; scoring by model judges was not validated by human assessors; four authors are affiliated with ResearchHub Foundation, which is implementing AI-assisted review. The abstract refers to 12 models for timing while 15 were tested.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-29; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Methodological and reporting errors in clinical trial manuscripts slip past overloaded human reviewers.
- Stakeholders
- Research integrity officer
- Clinical research office director
- Biostatistics core lead
- University press or journal editors
- Discovery questions
- Which manuscript checks are run before submission today?
- Can draft manuscripts be sent to commercial models, or only local ones?
- Who would triage AI flags?
- Value hypothesis
- A validated AI check may catch checkable reporting errors before submission, with false flags handled by staff.
- Potential engagement
- An evaluation comparing several models on seeded and clean local manuscripts for detection, specificity and cost.
- Unsupported claims
- That AI replaces human peer review
- Performance beyond in-text clinical trial errors
- Superiority over trained human reviewers today
Pre-sales engineering
Role takeaway
- Fit
- Fits clinical research units and integrity offices handling trial reports.
- Architecture and integration
- Manuscript intake, model review with fixed guidance, blinded scoring against an answer key and flag reporting.
- Prerequisites
- Approved model service or local open-weight model
- Seeded test manuscripts
- Error answer key
- Constraints
- Small published test set
- Manuscript-only checks miss data and code errors
- Preprint results
- Security
- Keep unpublished manuscripts on approved or local models and log access.
- Proof of value
- Measure specificity on local clean manuscripts
- Have humans validate a sample of judge scores
- Compare cost per review across models
Delivery
Role takeaway
- Work
- Build local test sets, choose models, set up advisory reporting and train staff to triage flags.
- Dependencies
- Statistics expertise for answer keys
- Model approval
- Author consent for drafts
- Ownership
- Research integrity office or clinical research support unit.
- Skills and adoption
- Authors learn to read AI flags critically; staff learn to separate real errors from interpretive comments.
- Governance checkpoints
- Clear AI-generated labelling
- Human decision on every flag
- Periodic re-testing after model updates
- Acceptance criteria
- Detection and specificity above agreed thresholds on local tests (proposal)
- Flag triage within an agreed time
- No manuscripts sent to unapproved services
- Risks
- False flags burdening authors
- Authors gaming the screen
- Shared biases in model judges
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Manuscript-only checks are limited; the authors suggest giving reviewers, human or AI, access to data, code, protocols and registry entries.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports AI checks as a complement with clear labelling, not a replacement; false flags on controls require human triage.
Security and privacy
What data, permissions and controls need testing?
Open-weight models that run locally scored 66% to 87%, an option where manuscripts cannot go to commercial services.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Could reduce reviewer burden on checkable reporting errors so experts focus on judgment.
Procurement
What should contracts, pricing and exit terms secure?
Cost and detection varied widely and did not track each other; buyers should compare models on their own error sets.
Operating model
Which teams own the service once it runs?
Checks could run at preprint submission or during manuscript preparation, with outputs published alongside.
Publication history
- 2026-10-29Research Universities · Issue 514 resources
Stable resource ID: medrxiv-llms-detect-clinical-trial-methodological-errors-20261028