From the Research Universities edition of September 20, 2026
ClaroAI-Bench: full-capability agent reproduces 20/33 NIH papers (60.6%); audit/bash-only hit 0% D5
O'Connell et al. (bioRxiv; correspondence notes Deloitte affiliation) · Biomedical computational reproducibility / AI agent evaluation · United States (NIH-funded paper corpus); transferable with biomedical data-access limits
- Publisher
- bioRxiv
- Original publication
- May 12, 2026 (bioRxiv posted)
- Source retrieved
- 2026-09-21
What happened
ClaroAI-Bench evaluates agents on 35 real NIH-funded biomedical papers across genomics, imaging, clinical/EHR, epidemiology, and wet-lab modalities using a five-dimension rubric (D1–D5). In a three-condition ablation, audit-only and bash-only agents achieve 0% D5 results reproduction, while a full-capability agent (Claude Code, all tools) reproduces 20 of 33 computational papers (60.6%; 95% CI [42.4, 75.8]). D1–D4 metadata scores predict D5 (Spearman r=0.68, p<0.0001), and papers with accessible data and code achieve 2.9× higher D5 scores than restricted papers (p=0.0013); environment reconstruction (D4) shows the highest inter-model scoring disagreement.
Why it matters
Why now: Newly relevant after the 2026-09-19 ML-focused process/certification benches—ClaroAI tests end-to-end biomedical reproduction under real data/code/environment access constraints that campus research computing and library data services must plan for, distinguishing tool-rich agents from metadata audits.
Evidence and measured results
35 NIH papers; D1–D5 rubric; 0% D5 for audit-only and bash-only; 20/33 (60.6%, CI [42.4, 75.8]); Spearman r=0.68; 2.9× access effect; D4 inter-model disagreement range cited in abstract.
Limitations and uncertainty
May 2026 preprint older than last run—retained with why-now; single full-capability agent condition; computational subset 33/35; correspondence affiliation includes Deloitte; not multi-campus production.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-20; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Institutions cannot tell whether AI agents can reproduce biomedical computational findings when data, code, and environments are incomplete or restricted.
- Stakeholders
- Biomedical PIs
- Research computing
- Library/data services
- Compliance/IRB
- Discovery questions
- Which papers have accessible data and code?
- Where do environment rebuilds fail?
- What tool scopes are allowed on controlled data?
- How is D5 verified against published claims?
- Value hypothesis
- Tool-rich agents can reproduce a substantial minority-to-majority of computational NIH papers (20/33, 60.6% in this study) when access exists—while audit-only/bash-only paths fail D5—if data accessibility is treated as first-class infrastructure.
- Potential engagement
- Bounded reproduction pilot on N papers with access-tier tagging and D1–D5 scorecards.
- Unsupported claims
- Guaranteed reproduction of all NIH papers
- Metadata audit equals results reproducibility
- Safe unrestricted agent tools on EHR data
Pre-sales engineering
Role takeaway
- Fit
- End-to-end biomedical reproduction agent evaluation with D1–D5 rubric and access-aware ablations.
- Architecture and integration
- Connect data catalogs, credential brokers, env builders, and compute runners; log every access and execution step.
- Prerequisites
- Data access pathways
- Code availability tracking
- Reproducible environment specs
- Results verification harness
- Constraints
- Single full-capability agent in reported ablation
- Preprint May 2026
- Computational paper subset
- Security
- Scope credentials; prohibit unconstrained tool use on controlled clinical/genomic data; retain audit trails.
- Proof of value
- D5 rate with CI
- Ablation vs audit-only/bash-only
- Correlation of D1–D4 with D5 on local corpus
Delivery
Role takeaway
- Work
- Select N papers, tag access tiers, run audit-only and tool-rich conditions, score D1–D5, and remediate top access blockers.
- Dependencies
- Data stewards
- Research computing
- Compliance
- PI scientific owners
- Ownership
- Evaluation lead owns scoring; data stewards own access; PI owns claim acceptance.
- Skills and adoption
- Train analysts on env reconstruction and restricted-data agent operations.
- Governance checkpoints
- Data-use approval
- Credential scope review
- D5 acceptance sign-off
- Acceptance criteria
- Documented D5 outcomes
- Access-tier inventory
- Ablation record
- Risks
- Sensitive-data tool sprawl
- Overgeneralizing 60.6%
- Ignoring D4 scoring disagreement
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Biomedical reproduction agents need data catalogs, secrets/credential brokers, and environment builders—not bash alone; restricted data paths dominate failure modes.
Governance
Who approves, reviews and stays accountable for outcomes?
Prioritize data/code accessibility policies if institutions expect AI-assisted reproduction; metadata audits are insufficient acceptance gates.
Security and privacy
What data, permissions and controls need testing?
Full-capability agents with broad tools amplify risk on clinical/EHR and controlled datasets—require scoped credentials and audit logs.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Accessible data/code strongly condition reproduction success; workforce needs data librarians and RSEs, not only agent operators.
Procurement
What should contracts, pricing and exit terms secure?
Fund data accessibility and environment packaging alongside agent tools; reject procurements that only score metadata audits.
Operating model
Which teams own the service once it runs?
Makes data stewardship and env engineering co-equal with agent platforms in the research-AI operating model.
Publication history
- 2026-09-20Research Universities · Issue 154 resources
Stable resource ID: claroai-bench-biomedical-reproducibility-20260512