{"resourceId":"claroai-bench-biomedical-reproducibility-20260512","versions":[{"version":"external-1e77418a136a9c3a425ded913bfc79ec2940206c5d6ef2054e3c0bbab44c140b","resource":{"id":"claroai-bench-biomedical-reproducibility-20260512","title":"ClaroAI-Bench: full-capability agent reproduces 20/33 NIH papers (60.6%); audit/bash-only hit 0% D5","organization":"O'Connell et al. (bioRxiv; correspondence notes Deloitte affiliation)","sector":"Biomedical computational reproducibility / AI agent evaluation","geography":"United States (NIH-funded paper corpus); transferable with biomedical data-access limits","publishedAt":"May 12, 2026 (bioRxiv posted)","publicationDate":"2026-05-12","eventDate":null,"sourceName":"bioRxiv","sourceLabel":"Academic preprint — ClaroAI-Bench biomedical agentic reproducibility suite","sourceUrl":"https://www.biorxiv.org/content/10.64898/2026.05.08.723611v1.full","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","data-security","infrastructure","governance-procurement","operating-model"],"finding":"ClaroAI-Bench evaluates agents on 35 real NIH-funded biomedical papers across genomics, imaging, clinical/EHR, epidemiology, and wet-lab modalities using a five-dimension rubric (D1–D5). In a three-condition ablation, audit-only and bash-only agents achieve 0% D5 results reproduction, while a full-capability agent (Claude Code, all tools) reproduces 20 of 33 computational papers (60.6%; 95% CI [42.4, 75.8]). D1–D4 metadata scores predict D5 (Spearman r=0.68, p<0.0001), and papers with accessible data and code achieve 2.9× higher D5 scores than restricted papers (p=0.0013); environment reconstruction (D4) shows the highest inter-model scoring disagreement.","sledRelevance":"Why now: Newly relevant after the 2026-09-19 ML-focused process/certification benches—ClaroAI tests end-to-end biomedical reproduction under real data/code/environment access constraints that campus research computing and library data services must plan for, distinguishing tool-rich agents from metadata audits.","evidence":"35 NIH papers; D1–D5 rubric; 0% D5 for audit-only and bash-only; 20/33 (60.6%, CI [42.4, 75.8]); Spearman r=0.68; 2.9× access effect; D4 inter-model disagreement range cited in abstract.","architectureImplications":"Biomedical reproduction agents need data catalogs, secrets/credential brokers, and environment builders—not bash alone; restricted data paths dominate failure modes.","governanceImplications":"Prioritize data/code accessibility policies if institutions expect AI-assisted reproduction; metadata audits are insufficient acceptance gates.","securityPrivacyImplications":"Full-capability agents with broad tools amplify risk on clinical/EHR and controlled datasets—require scoped credentials and audit logs.","caveats":"May 2026 preprint older than last run—retained with why-now; single full-capability agent condition; computational subset 33/35; correspondence affiliation includes Deloitte; not multi-campus production.","streamIds":["research"],"roles":{"sales":"Sales framing: Biomedical research leaders want AI that can actually re-run published analyses under real access constraints—not another coding demo on toy repos. ClaroAI-Bench evidence supports a bounded engagement around end-to-end reproduction tooling and data-access remediation (20/33 = 60.6% for a full-capability agent; 0% for audit-only and bash-only)—not a guarantee every NIH paper will reproduce. Discovery should map which papers have usable data/code and where environment rebuilds fail. Unsupported: campus-wide autonomous reproduction, guaranteed 60%+ on restricted corpora, or that metadata scores alone prove D5 success.","engineering":"Engineering framing: Fit is a long-horizon agent evaluation spanning data findability/access, code obtainment, environment reconstruction, execution, and results verification. Integration points are institutional data catalogs, credential brokers, container registries, and HPC/cloud runners. Prerequisites: clear data-access paths, reconstructable environments, and logging. Constraints: preprint; one full-capability agent condition; biomedical modalities. Security must scope agent credentials on EHR/genomic data and retain provenance. Proof of value is D5 success with CI plus ablation showing tool/access dependence.","delivery":"Delivery framing: Pilot reproduction on a small set of NIH-funded computational papers with known access tiers, comparing audit-only vs tool-rich agents under human oversight. Dependencies: library/data services, research computing, compliance, and PI owners. Ownership: evaluation lead owns rubric scoring; data stewards own access; PI owns scientific acceptance. Skills adoption trains staff on environment rebuild and restricted-data workflows. Governance checkpoints: data-use approval, credential scoping, and D5 acceptance criteria. Acceptance: documented D5 outcomes with access metadata; risks include over-tooling agents on sensitive data and ignoring D4 disagreement."},"roleDetails":{"sales":{"customerProblem":"Institutions cannot tell whether AI agents can reproduce biomedical computational findings when data, code, and environments are incomplete or restricted.","stakeholders":["Biomedical PIs","Research computing","Library/data services","Compliance/IRB"],"discoveryQuestions":["Which papers have accessible data and code?","Where do environment rebuilds fail?","What tool scopes are allowed on controlled data?","How is D5 verified against published claims?"],"valueHypothesis":"Tool-rich agents can reproduce a substantial minority-to-majority of computational NIH papers (20/33, 60.6% in this study) when access exists—while audit-only/bash-only paths fail D5—if data accessibility is treated as first-class infrastructure.","potentialEngagement":"Bounded reproduction pilot on N papers with access-tier tagging and D1–D5 scorecards.","unsupportedClaims":["Guaranteed reproduction of all NIH papers","Metadata audit equals results reproducibility","Safe unrestricted agent tools on EHR data"]},"engineering":{"fit":"End-to-end biomedical reproduction agent evaluation with D1–D5 rubric and access-aware ablations.","architectureIntegration":"Connect data catalogs, credential brokers, env builders, and compute runners; log every access and execution step.","prerequisites":["Data access pathways","Code availability tracking","Reproducible environment specs","Results verification harness"],"constraints":["Single full-capability agent in reported ablation","Preprint May 2026","Computational paper subset"],"security":"Scope credentials; prohibit unconstrained tool use on controlled clinical/genomic data; retain audit trails.","proofOfValue":["D5 rate with CI","Ablation vs audit-only/bash-only","Correlation of D1–D4 with D5 on local corpus"]},"delivery":{"work":"Select N papers, tag access tiers, run audit-only and tool-rich conditions, score D1–D5, and remediate top access blockers.","dependencies":["Data stewards","Research computing","Compliance","PI scientific owners"],"ownership":"Evaluation lead owns scoring; data stewards own access; PI owns claim acceptance.","skillsAdoption":"Train analysts on env reconstruction and restricted-data agent operations.","governanceCheckpoints":["Data-use approval","Credential scope review","D5 acceptance sign-off"],"acceptanceCriteria":["Documented D5 outcomes","Access-tier inventory","Ablation record"],"risks":["Sensitive-data tool sprawl","Overgeneralizing 60.6%","Ignoring D4 scoring disagreement"]}},"retrievedAt":"2026-09-20T22:11:20-05:00","enrichedAt":"2026-09-20T22:11:20-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Accessible data/code strongly condition reproduction success; workforce needs data librarians and RSEs, not only agent operators.","procurementImplications":"Fund data accessibility and environment packaging alongside agent tools; reject procurements that only score metadata audits.","operatingModelImplications":"Makes data stewardship and env engineering co-equal with agent platforms in the research-AI operating model.","sourceVerification":{"openedUrl":"https://www.biorxiv.org/content/10.64898/2026.05.08.723611v1.full","referenceExcerpt":"reproduces 20 of 33 computational papers (60.6%; 95% CI [42.4, 75.8])","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}