From the Research Universities edition of October 9, 2026
Nature Medicine: LLM-assisted review finds only 12.2% of 3,967 clinical prediction-model studies share code, and shared repositories often lack reproducibility features
Sounack et al. (TRIPOD-Code initiative), Nature Medicine · Academic research reproducibility / research integrity · International literature (PubMed Central Open Access)
- Publisher
- Nature Medicine
- Original publication
- Nature Medicine, Analysis, published 9 October 2026 (received 28 Mar, accepted 1 Sep 2026)
- Source retrieved
- 2026-10-10
What happened
An LLM-assisted pipeline screened articles citing TRIPOD or TRIPOD+AI, extracted repository links and scored repositories against 14 reproducibility features. Of 3,967 articles, 482 (12.2%) had code-sharing statements; prevalence varied widely by journal and country; repositories often lacked documentation, dependency specification, licensing and executable structure. Pipeline validation on 500 manually annotated articles: weighted F1 0.97 for selection, 92.3% accuracy on repository-link retrieval.
Why it matters
Two research-office signals: code availability alone does not deliver reproducibility, and LLM-assisted pipelines can audit literature at scale with human-validated accuracy, an approach research integrity and library units could adapt.
Evidence and measured results
Opened the open-access Nature Medicine article via HTTPS (curl, full text extracted); read abstract, methods (validation against 500 human-annotated articles), results and limitations.
Limitations and uncertainty
Authors note LLM misclassification may affect point estimates; cohort limited to TRIPOD-citing PMC Open Access papers, likely overestimating sharing; may not generalize to other checklists (STARD, CLAIM); clinical domain only.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-09; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Institutions mandate or encourage code sharing but have no evidence that shared code is reusable or reproducible.
- Stakeholders
- Research integrity offices
- Academic medical center research leadership
- University libraries/research data services
- Research software engineering groups
- Discovery questions
- How does the institution assess reproducibility of deposited code today?
- Would it audit its own publications with an LLM-assisted screen?
- What repository features are required at deposit?
- Value hypothesis
- A validated automated screen could surface reproducibility gaps across an institution's output faster than manual review; accuracy must be re-validated locally.
- Potential engagement
- Six-week pilot applying an LLM-assisted code-sharing audit to one school's publications with a human-annotated validation sample.
- Unsupported claims
- That LLM screening replaces human review
- That code-sharing mandates alone ensure reproducibility
- Results generalize beyond clinical prediction models
Pre-sales engineering
Role takeaway
- Fit
- Research integrity or library teams auditing publication and repository practices at scale.
- Architecture and integration
- Publication list ingestion, open-access full-text retrieval, repository fetch and LLM-based feature scoring with human-labelled validation set.
- Prerequisites
- Access to full text
- Human-annotated validation sample
- Repository API access
- Constraints
- Misclassification affects point estimates
- Restricted to open-access papers
- Clinical domain specificity
- Security
- Screen repositories for embedded patient data or secrets and keep LLM processing within approved data boundaries.
- Proof of value
- Reproduce F1 on a local 200-article sample
- Compare automated feature scores to manual review on 50 repositories
- Measure false-positive rate on link retrieval
Delivery
Role takeaway
- Work
- Define required repository features, deploy screening with human review, and train researchers on reproducible packaging.
- Dependencies
- Institutional policy on code sharing
- Repository platform
- Research software engineering support
- Ownership
- Research integrity office with library research data services.
- Skills and adoption
- Provide templates and workshops on dependency files, licensing and run instructions.
- Governance checkpoints
- Validation results reviewed before reporting
- Policy approval by research council
- Periodic audit of screening accuracy
- Acceptance criteria
- Proposed: 80% of new deposits include dependency specification
- Proposed: local validation F1 at least 0.9
- Proposed: quarterly audit report delivered
- Risks
- Researcher burden
- Automated misclassification
- Perceived surveillance
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Institutional repositories and research software services could enforce dependency files, licenses and run instructions at deposit; LLM-assisted screening needs a human validation set.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports institutional or funder expectations beyond 'code available' (documentation, dependencies, licensing), feeding the planned TRIPOD-Code guideline.
Security and privacy
What data, permissions and controls need testing?
Clinical prediction code may embed patient-data paths or credentials; repository review policies should check for leakage when mandating sharing (interpretation).
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Researchers need training in packaging reproducible code; research software engineering support is implied.
Procurement
What should contracts, pricing and exit terms secure?
Limited; LLM-assisted review tooling could be procured or built by research integrity offices.
Operating model
Which teams own the service once it runs?
Moves reproducibility checking from manual spot checks toward automated, validated screening with human oversight.
Publication history
- 2026-10-09Research Universities · Issue 324 resources
Stable resource ID: nature-medicine-code-sharing-clinical-prediction-models-20261009