{"resourceId":"northwestern-autonomous-science-bench-budgeted-campaigns-20261013","versions":[{"version":"external-0428d74b4c5bc87a05a1c468436ccf5bb195c0174e48feaf43cfabd61f5bde10","resource":{"id":"northwestern-autonomous-science-bench-budgeted-campaigns-20261013","title":"Autonomous Science Bench (Northwestern preprint): agents pass only 31 of 144 budgeted lab campaigns, and the best setup costs 25 times the cheapest","organization":"Northwestern University (Wen and Liu, arXiv cs.AI)","sector":"Higher education research and research agents","geography":"United States; campaigns use published experiments from international labs","publishedAt":"13 October 2026 (arXiv v1)","publicationDate":"2026-10-13","eventDate":null,"sourceName":"arXiv","sourceLabel":"arXiv preprint (v1), not peer reviewed","sourceUrl":"https://arxiv.org/abs/2610.18449","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","governance-procurement"],"finding":"Autonomous Science Bench v0.1 gives agents six budgeted design-build-test-learn campaigns (two each in biology, chemistry and materials) built on published experiments, with a lab service that returns measurements and keeps a ledger. A campaign passes only if both a discovery gate and a learning gate pass. In 144 trials across eight agent-model configurations, only 31 passed; the best configuration resolved 67% of its trials and the hardest campaign was solved in three of 24 attempts. Higher success came at steeply higher cost: total cost rose 25-fold from $6.91 to $172.90 between the cheapest and best configurations, and cost per resolved trial from $1.38 to $14.41. Agents failed in different ways: some ranked well but found too few hits, others found hits but could not predict unseen designs.","sledRelevance":"Research offices and self-driving-lab programmes need ways to judge agent experiment planning before committing instrument time. This benchmark shows success, failure modes and cost vary widely by configuration and campaign.","evidence":"Opened the arXiv abstract page and the full HTML paper: campaign table, experimental setup, cost and performance results, failure analysis and limitations.","architectureImplications":"A lab service with an append-only ledger, fixed budgets and separate discovery and learning checks is a usable pattern for evaluating agents before connecting them to instruments.","governanceImplications":"Overall success rates hide distinct failure modes, so acceptance tests for agent-run experimentation should check both what was found and what the agent learned.","securityPrivacyImplications":"Not addressed beyond egress controls to stop agents retrieving published answers during trials.","caveats":"Small pilot (six campaigns, eight configurations); replay and digital-twin backends, not live labs; public data may be memorised by models; preprint not peer reviewed; cost figures are API costs for these runs only.","streamIds":["research"],"roles":{"sales":"Lead with cost per verified result. In this benchmark, agents passed 31 of 144 lab campaigns, and the best setup cost 25 times the cheapest while resolving 2.4 times as many trials. Directors of self-driving labs and research computing should hear that agent choice is a cost and reliability trade-off. Interpretation: this is a small preprint on replayed data, supporting an evaluation step before purchase, not promised discovery rates.","engineering":"Use the benchmark design as an evaluation harness. It pairs a stateful lab service and ledger with fixed budgets and two gates, so failures in discovery and in learning show separately. Pre-sales can propose running candidate agents on replayed campus data before instrument access. Interpretation: results come from replay and twin backends, and public data may be memorised, so live-lab performance is untested.","delivery":"Frame delivery as a staged gate from replay to live lab. Teams would build budgeted replay campaigns from their own past experiments, measure both gates and cost per resolved trial, and only then connect agents to instruments. The lab or core facility director owns the gate. Interpretation: acceptance thresholds should be set per campaign, since difficulty varied sharply."},"roleDetails":{"sales":{"customerProblem":"Labs considering agent-driven experimentation cannot compare agents on reliability and cost before committing instrument time.","stakeholders":["Self-driving lab or core facility director","Research computing director","Department chairs in chemistry, biology and materials","Procurement"],"discoveryQuestions":["Do you have past experiments that could be replayed to test an agent?","How do you compare agent tools on cost per verified outcome?","Who decides when an agent may run live instruments?"],"valueHypothesis":"An evaluation on replayed campus campaigns may avoid committing instrument time and budget to agents that fail on local problems.","potentialEngagement":"A short benchmark study running two or three agent configurations on replayed local campaigns, reporting pass rates and cost per resolved trial.","unsupportedClaims":["Agents reliably run experimental campaigns","Live-lab performance matches benchmark results","Fixed cost savings from agent experimentation"]},"engineering":{"fit":"Fits labs with recorded experimental data and plans for autonomous experimentation; limited to small pilot scope.","architectureIntegration":"Wrap past experiment data in a budgeted lab service with a ledger and score agents on separate discovery and learning gates.","prerequisites":["Recorded experiment datasets with outcomes","Agent platforms to compare","Network egress controls during trials"],"constraints":["Replay and twin backends only","Possible model memorisation of published data","Six campaigns in three fields"],"security":"Block agent internet access during trials to stop answer retrieval; otherwise not addressed.","proofOfValue":["Run repeated trials per configuration on a local replay campaign","Report cost per resolved trial","Check both gates, not overall pass rate"]},"delivery":{"work":"Build replay campaigns from local experiments, run agent evaluations, and set a go or no-go gate for live instrument access.","dependencies":["Data stewards for past experiments","Agent platform access","Budget for API costs"],"ownership":"Core facility or self-driving lab director, with research computing support.","skillsAdoption":"Staff learn to construct campaigns, budgets and gates, and to read agent ledgers.","governanceCheckpoints":["Gate review before live instrument access","Cost review per campaign","Repeat-trial review"],"acceptanceCriteria":["Proposal: agent passes both gates in a set share of repeated replay trials before live use","Proposal: cost per resolved trial reported for each configuration","Proposal: all agent measurement requests recorded in a ledger"],"risks":["Benchmark results not transferring to live labs","Cost overruns with higher-capability models","Choosing agents on overall pass rate alone"]}},"retrievedAt":"2026-10-18T21:59:00-05:00","enrichedAt":"2026-10-18T21:59:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Expert-time estimates for each campaign are provided, but the paper does not measure staff time saved.","procurementImplications":"Cost per resolved trial differed tenfold across configurations; buyers should compare cost per verified outcome, not per run or token.","operatingModelImplications":"Agent experimentation should be piloted on replayed or simulated campaigns with budgets before live instrument access.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.18449","referenceExcerpt":"the best configuration succeeds in 67% of trials, while the hardest campaign is solved in only three of 24 attempts","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}