{"resourceId":"medrxiv-llms-detect-clinical-trial-methodological-errors-20261028","versions":[{"version":"external-1a8840ef1399d7586d36b7e0d55ff732f6806ec9fbde797e5d632d6381f9de96","resource":{"id":"medrxiv-llms-detect-clinical-trial-methodological-errors-20261028","title":"Preprint: 15 LLMs found 51% to 99% of errors inserted into 2026 clinical trial reports, versus 27% for untrained human reviewers in an earlier study, but also flagged problems in unaltered controls","organization":"ResearchHub Foundation, University of Southern California, Western University and OrganoBit BioIntelligence","sector":"Scholarly peer review and research integrity (clinical research)","geography":"United States, Canada and Singapore (authors); international journals","publishedAt":"28 October 2026 (medRxiv posting)","publicationDate":"2026-10-28","eventDate":null,"sourceName":"medRxiv","sourceLabel":"Brian, Pereira Hernández, Yin, Koury and Rust, \"Large language models detect methodological errors that human peer review misses in clinical trials,\" medRxiv, posted 28 Oct 2026","sourceUrl":"https://www.medrxiv.org/content/medrxiv/early/2026/10/28/2026.10.25.26366455.full.pdf","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","governance-procurement"],"finding":"The authors repeated the largest inserted-error peer review experiment, in which 607 untrained human reviewers found a mean of 27% of major errors in three trial reports, with 15 LLMs from six developers, including four open-weight models. On the original papers, LLM major-error detection scores ranged from 53% to 96%. To reduce memorization, they inserted the same 14 error types into three randomized trials published in 2026: scores ranged from 51% to 99% (66% to 87% for open-weight models), with Opus 5.5 highest at 99%. On unmodified control versions, models flagged 2% to 29% of error types (overall specificity 83.0%), and models that found more errors also flagged more on controls (Spearman rho -0.77). Blinded model judges scored reviews with 94.4% and 97.9% agreement. Review times were 1.4 to 16.3 minutes and estimated costs $0.005 to $2.24 per review for the 12 models timed. The authors recommend AI-assisted methodological checks as a complement to human review.","sledRelevance":"Research integrity offices, clinical research units and university presses considering AI pre-submission checks get a controlled benchmark that includes false-positive rates on clean manuscripts. It contrasts with recent cautionary studies of AI review in this stream, showing narrow, checkable tasks fare better than overall judgments.","evidence":"Read the abstract, results, discussion, limitations and competing interests in the full-text PDF. Supplementary tables were not opened. The human comparison is historical.","architectureImplications":"Manuscript-only checks are limited; the authors suggest giving reviewers, human or AI, access to data, code, protocols and registry entries.","governanceImplications":"Supports AI checks as a complement with clear labelling, not a replacement; false flags on controls require human triage.","securityPrivacyImplications":"Open-weight models that run locally scored 66% to 87%, an option where manuscripts cannot go to commercial services.","caveats":"Preprint, not peer reviewed. Small set of six trial reports and predefined, in-text error types; human comparison uses historical reviewers on different manuscripts; scoring by model judges was not validated by human assessors; four authors are affiliated with ResearchHub Foundation, which is implementing AI-assisted review. The abstract refers to 12 models for timing while 15 were tested.","streamIds":["research"],"roles":{"sales":"Lead with narrow, checkable quality control rather than AI peer review. A 28 October medRxiv preprint found 15 LLMs detected 51% to 99% of errors inserted into 2026 clinical trial reports, against 27% for untrained human reviewers in an earlier study, while also flagging issues in clean controls. Clinical research and integrity offices may want pre-submission checks. Offer a bounded evaluation. Do not claim AI replaces reviewers; this is an unreviewed preprint with industry-affiliated authors.","engineering":"Frame fit around a pre-submission methodological check with measured false positives. The study seeded known errors, kept unmodified controls and scored reviews with blinded model judges. An institution can reuse that design with its own manuscripts. Prerequisites are an approved model, ideally open-weight for sensitive drafts, and an error answer key. Validate specificity on local clean manuscripts and have humans check judge scores, which the authors did not.","delivery":"Treat delivery as an advisory checking service for draft manuscripts. Work covers building local seeded test sets, selecting models on detection, specificity and cost, and routing flags to authors with clear labels. Ownership sits with the research integrity office or clinical research support unit. Governance checkpoints keep authors and reviewers accountable. Acceptance criteria are proposals, such as agreed detection and specificity thresholds on local tests, not results reported here."},"roleDetails":{"sales":{"customerProblem":"Methodological and reporting errors in clinical trial manuscripts slip past overloaded human reviewers.","stakeholders":["Research integrity officer","Clinical research office director","Biostatistics core lead","University press or journal editors"],"discoveryQuestions":["Which manuscript checks are run before submission today?","Can draft manuscripts be sent to commercial models, or only local ones?","Who would triage AI flags?"],"valueHypothesis":"A validated AI check may catch checkable reporting errors before submission, with false flags handled by staff.","potentialEngagement":"An evaluation comparing several models on seeded and clean local manuscripts for detection, specificity and cost.","unsupportedClaims":["That AI replaces human peer review","Performance beyond in-text clinical trial errors","Superiority over trained human reviewers today"]},"engineering":{"fit":"Fits clinical research units and integrity offices handling trial reports.","architectureIntegration":"Manuscript intake, model review with fixed guidance, blinded scoring against an answer key and flag reporting.","prerequisites":["Approved model service or local open-weight model","Seeded test manuscripts","Error answer key"],"constraints":["Small published test set","Manuscript-only checks miss data and code errors","Preprint results"],"security":"Keep unpublished manuscripts on approved or local models and log access.","proofOfValue":["Measure specificity on local clean manuscripts","Have humans validate a sample of judge scores","Compare cost per review across models"]},"delivery":{"work":"Build local test sets, choose models, set up advisory reporting and train staff to triage flags.","dependencies":["Statistics expertise for answer keys","Model approval","Author consent for drafts"],"ownership":"Research integrity office or clinical research support unit.","skillsAdoption":"Authors learn to read AI flags critically; staff learn to separate real errors from interpretive comments.","governanceCheckpoints":["Clear AI-generated labelling","Human decision on every flag","Periodic re-testing after model updates"],"acceptanceCriteria":["Detection and specificity above agreed thresholds on local tests (proposal)","Flag triage within an agreed time","No manuscripts sent to unapproved services"],"risks":["False flags burdening authors","Authors gaming the screen","Shared biases in model judges"]}},"retrievedAt":"2026-10-29T22:01:29-05:00","enrichedAt":"2026-10-29T22:01:33-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Could reduce reviewer burden on checkable reporting errors so experts focus on judgment.","procurementImplications":"Cost and detection varied widely and did not track each other; buyers should compare models on their own error sets.","operatingModelImplications":"Checks could run at preprint submission or during manuscript preparation, with outputs published alongside.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://www.medrxiv.org/content/medrxiv/early/2026/10/28/2026.10.25.26366455.full.pdf","referenceExcerpt":"Notably, LLMs still flagged 2% to 29% of the error types in control reports without inserted errors","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}