{"resourceId":"nature-medicine-code-sharing-clinical-prediction-models-20261009","versions":[{"version":"external-6f48e8d322b2dc2d130d60ae02548e154f14250e93921721c6a80c0f53c1d2e2","resource":{"id":"nature-medicine-code-sharing-clinical-prediction-models-20261009","title":"Nature Medicine: LLM-assisted review finds only 12.2% of 3,967 clinical prediction-model studies share code, and shared repositories often lack reproducibility features","organization":"Sounack et al. (TRIPOD-Code initiative), Nature Medicine","sector":"Academic research reproducibility / research integrity","geography":"International literature (PubMed Central Open Access)","publishedAt":"Nature Medicine, Analysis, published 9 October 2026 (received 28 Mar, accepted 1 Sep 2026)","publicationDate":"2026-10-09","eventDate":null,"sourceName":"Nature Medicine","sourceLabel":"Peer-reviewed analysis (open access)","sourceUrl":"https://www.nature.com/articles/s41591-026-04691-1","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","governance-procurement","developers-agents"],"finding":"An LLM-assisted pipeline screened articles citing TRIPOD or TRIPOD+AI, extracted repository links and scored repositories against 14 reproducibility features. Of 3,967 articles, 482 (12.2%) had code-sharing statements; prevalence varied widely by journal and country; repositories often lacked documentation, dependency specification, licensing and executable structure. Pipeline validation on 500 manually annotated articles: weighted F1 0.97 for selection, 92.3% accuracy on repository-link retrieval.","sledRelevance":"Two research-office signals: code availability alone does not deliver reproducibility, and LLM-assisted pipelines can audit literature at scale with human-validated accuracy, an approach research integrity and library units could adapt.","evidence":"Opened the open-access Nature Medicine article via HTTPS (curl, full text extracted); read abstract, methods (validation against 500 human-annotated articles), results and limitations.","architectureImplications":"Institutional repositories and research software services could enforce dependency files, licenses and run instructions at deposit; LLM-assisted screening needs a human validation set.","governanceImplications":"Supports institutional or funder expectations beyond 'code available' (documentation, dependencies, licensing), feeding the planned TRIPOD-Code guideline.","securityPrivacyImplications":"Clinical prediction code may embed patient-data paths or credentials; repository review policies should check for leakage when mandating sharing (interpretation).","caveats":"Authors note LLM misclassification may affect point estimates; cohort limited to TRIPOD-citing PMC Open Access papers, likely overestimating sharing; may not generalize to other checklists (STARD, CLAIM); clinical domain only.","streamIds":["research"],"roles":{"sales":"Sales framing: research offices and academic medical centers face evidence that shared code rarely reproduces. Ask how the institution checks reproducibility of deposited code and whether it audits its own literature. A bounded pilot applying a validated LLM-assisted screen to an institution's publications fits. The study cannot support claims that LLM review replaces human assessment or that sharing mandates alone fix reproducibility.","engineering":"Engineering framing: the pipeline pattern, LLM screening validated against 500 human labels with F1 0.97, is reusable for institutional audits. Integration means pulling publication lists, retrieving repositories and scoring features like dependencies and licenses. Prerequisites are open-access full text and a local validation set. Constraints include misclassification and domain specificity. Prove value by reproducing the validation step on local data before trusting aggregate figures.","delivery":"Delivery framing: turning this into practice means adding reproducibility checks to deposit and publication workflows. Delivery would define required repository features, set up automated screening with human review, and train researchers. Ownership sits with research integrity or library research data services. Propose acceptance measures such as the share of new deposits with dependency files; risks include researcher burden and false positives from automated screening."},"roleDetails":{"sales":{"customerProblem":"Institutions mandate or encourage code sharing but have no evidence that shared code is reusable or reproducible.","stakeholders":["Research integrity offices","Academic medical center research leadership","University libraries/research data services","Research software engineering groups"],"discoveryQuestions":["How does the institution assess reproducibility of deposited code today?","Would it audit its own publications with an LLM-assisted screen?","What repository features are required at deposit?"],"valueHypothesis":"A validated automated screen could surface reproducibility gaps across an institution's output faster than manual review; accuracy must be re-validated locally.","potentialEngagement":"Six-week pilot applying an LLM-assisted code-sharing audit to one school's publications with a human-annotated validation sample.","unsupportedClaims":["That LLM screening replaces human review","That code-sharing mandates alone ensure reproducibility","Results generalize beyond clinical prediction models"]},"engineering":{"fit":"Research integrity or library teams auditing publication and repository practices at scale.","architectureIntegration":"Publication list ingestion, open-access full-text retrieval, repository fetch and LLM-based feature scoring with human-labelled validation set.","prerequisites":["Access to full text","Human-annotated validation sample","Repository API access"],"constraints":["Misclassification affects point estimates","Restricted to open-access papers","Clinical domain specificity"],"security":"Screen repositories for embedded patient data or secrets and keep LLM processing within approved data boundaries.","proofOfValue":["Reproduce F1 on a local 200-article sample","Compare automated feature scores to manual review on 50 repositories","Measure false-positive rate on link retrieval"]},"delivery":{"work":"Define required repository features, deploy screening with human review, and train researchers on reproducible packaging.","dependencies":["Institutional policy on code sharing","Repository platform","Research software engineering support"],"ownership":"Research integrity office with library research data services.","skillsAdoption":"Provide templates and workshops on dependency files, licensing and run instructions.","governanceCheckpoints":["Validation results reviewed before reporting","Policy approval by research council","Periodic audit of screening accuracy"],"acceptanceCriteria":["Proposed: 80% of new deposits include dependency specification","Proposed: local validation F1 at least 0.9","Proposed: quarterly audit report delivered"],"risks":["Researcher burden","Automated misclassification","Perceived surveillance"]}},"retrievedAt":"2026-10-09T22:04:30-05:00","enrichedAt":"2026-10-09T22:07:06-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Researchers need training in packaging reproducible code; research software engineering support is implied.","procurementImplications":"Limited; LLM-assisted review tooling could be procured or built by research integrity offices.","operatingModelImplications":"Moves reproducibility checking from manual spot checks toward automated, validated screening with human oversight.","sourceVerification":{"openedUrl":"https://www.nature.com/articles/s41591-026-04691-1","referenceExcerpt":"Among 3,967 articles, 482 (12.2%) included code-sharing statements.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}