{"resourceId":"drugtargetworld-ai-scientist-biobank-benchmark-20261007","versions":[{"version":"external-6ad380ab4093023ae0e5041543627450e331ee4bb0174615e92c7d7e33487a54","resource":{"id":"drugtargetworld-ai-scientist-biobank-benchmark-20261007","title":"DrugTargetWorld (Stanford/Brown preprint): nine AI agents in 540 episodes average far below 40/100 on end-to-end drug-target discovery; leaders recover 64% of causal drivers but do not reliably reject non-causal proteins","organization":"Stanford University and Brown University (Margolis, Gomes and colleagues)","sector":"University biomedical AI / evaluation of AI scientists","geography":"United States (Stanford, CA; Brown University)","publishedAt":"arXiv:2610.09558v1, submitted 7 October 2026","publicationDate":"2026-10-07","eventDate":null,"sourceName":"arXiv (cs.AI)","sourceLabel":"Academic preprint (not peer-reviewed)","sourceUrl":"https://arxiv.org/abs/2610.09558","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work","data-security"],"finding":"DrugTargetWorld procedurally generates simulated cardiovascular biobanks ('worlds') with concealed causal structure—genotypes, proteins, health records, outcomes and synthetic MRI for 54,000 participants (10,800 with MRI)—so agents can be scored against known ground truth. Agents must build a disease phenotype, find causal driver proteins, infer beneficial modulation direction and optionally buy virtual wet-lab experiments. Across 540 episodes (20 worlds, three budgets, nine agents), Opus 5 and GPT-5.6 Sol led with mean composite scores 39.98 and 35.38 of 100; both recovered 64% of causal drivers on average, but no agent reliably distinguished misleading non-causal proteins. Phenotype construction earned zero credit in 416/540 episodes (77%). Mean total score across all episodes was 14.1/100. Authors argue agents can run many analyses but fail integrative judgments linking phenotype, causality and claim sufficiency.","sledRelevance":"Gives universities a concrete, verifiable benchmark showing AI 'scientists' still fail causal discrimination on biobank-style workflows—useful counterweight to campus AI infrastructure announcements that imply discovery automation is near.","evidence":"Opened the arXiv abstract page and full PDF (v1) via HTTPS (curl; pdftotext). Read abstract, introduction, evaluation design, results and limitations. Figures are authors' own; no independent replication.","architectureImplications":"Evaluating biomedical AI agents needs sandboxed code execution, episode budgets, sealed scoring keys and multimodal biobank-scale data—not only chat demos—so research computing must host agent eval harnesses separately from production PHI systems.","governanceImplications":"Interpretation: tool-selection and AI-scientist pilots should require causal-discrimination and bias-identification metrics, not only target-recall marketing numbers.","securityPrivacyImplications":"Synthetic worlds avoid participant-level access controls that block real biobank agent training; real deployments would reintroduce IRB, DUAs and PHI isolation the benchmark deliberately removes.","caveats":"Preprint; synthetic cardiovascular worlds only; transfer to real biobanks unproven; single run per model/world/budget so run-to-run variance not fully characterized; Opus 5 vs GPT-5.6 Sol paired difference not conclusive (P=0.062).","streamIds":["research"],"roles":{"sales":"Sales framing: campuses buying AI for biomedical discovery need proof agents can tell causal drivers from confounders. DrugTargetWorld's 540-episode eval shows top agents under 40/100 and failing non-causal rejection despite 64% driver recall. A bounded offer is an evaluation workshop that adopts causal-discrimination metrics for agent pilots. Do not claim agents are ready for unsupervised target nomination.","engineering":"Engineering framing: build agent evals with hidden ground truth, multimodal inputs, turn budgets and scored bias identification. Pin model versions, isolate sandboxes, and separate phenotype construction from final claim submission. Do not treat target recall alone as success when bias-identification scores stay near zero.","delivery":"Delivery framing: when piloting AI scientists with clinical or biobank partners, require phenotype validity checks and explicit rejection of non-causal candidates before any nomination leaves the lab. Biomedical informatics owns eval design; research computing owns sandbox; PIs own scientific acceptance. Interpretation: criteria below are proposals."},"roleDetails":{"sales":{"customerProblem":"Biomedical research leaders lack evaluations that test whether AI agents can distinguish causal drug targets from confounded associations.","stakeholders":["Biomedical data science leads","Research computing","Clinical research IT","VPR / research integrity"],"discoveryQuestions":["What metrics do you use today to accept an AI agent for target discovery support?","Do pilots score bias identification or only nomination recall?","Where would agent sandboxes run relative to PHI?"],"valueHypothesis":"Causal-discrimination benchmarks may prevent over-claiming agent readiness before real biobank pilots; no deployment benefit is reported.","potentialEngagement":"Six-week evaluation design sprint adapting DrugTargetWorld-style metrics to one local synthetic or de-identified cohort task set.","unsupportedClaims":["Agents are ready for unsupervised target nomination","64% driver recall equals clinical utility","Synthetic-world results transfer to real biobanks"]},"engineering":{"fit":"Labs evaluating code-executing AI agents on biomedical discovery workflows.","architectureIntegration":"Host sandboxed agent runners with turn limits, sealed scorers and multimodal data access; log analyses cited in final claims; keep real PHI out of training loops that need ground-truth causality.","prerequisites":["Sandbox capacity","Scoring harness","Model API governance"],"constraints":["Synthetic-only evidence","High zero-phenotype rates in the paper","Agent cost per episode"],"security":"Never expose real participant-level biobank data to agents without DUAs/IRB; prefer synthetic or heavily abstracted tasks for early evals.","proofOfValue":["Composite score on a frozen local task panel","Bias-identification score trend","Share of episodes with valid phenotype submissions"]},"delivery":{"work":"Define acceptance gates for AI-scientist pilots (phenotype validity, causal confidence, bias rejection) and run a small scored panel before any broader rollout.","dependencies":["Faculty evaluators","Research computing sandbox","Compliance review for any real data"],"ownership":"Biomedical informatics owns scientific metrics; research computing owns runtime; compliance owns data access.","skillsAdoption":"Train participating students/staff on reading agent trajectories and spotting unused analyses or unsupported claims.","governanceCheckpoints":["Written acceptance gates before pilot","Sandbox security review","Results reviewed by domain faculty"],"acceptanceCriteria":["Proposed: scored panel of ≥20 episodes on a fixed world set","Proposed: minimum bias-identification threshold set by faculty","Proposed: no unsupervised nominations released externally"],"risks":["Over-trust in leaderboard recall","Leakage of real PHI into agent logs","Pilot scope creep into clinical decisions"]}},"retrievedAt":"2026-10-09T13:16:11-05:00","enrichedAt":"2026-10-09T13:16:13-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Benchmarking end-to-end AI scientists requires multidisciplinary teams (cardiology, genetics, ML ops); smaller campuses may lack staff to run 540-episode-scale evals without shared harnesses.","procurementImplications":"When buying 'AI scientist' products, ask vendors for causal-discrimination and non-causal rejection rates under sealed evaluation, not only marketing demos.","operatingModelImplications":"Positions AI-scientist adoption as an evaluation and governance problem: campuses can expand GPU capacity while still requiring human integrative judgment for causal claims.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.09558","referenceExcerpt":"no agent reliably distinguished misleading non-causal proteins","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}