From the Research Universities edition of October 18, 2026
Autonomous Science Bench (Northwestern preprint): agents pass only 31 of 144 budgeted lab campaigns, and the best setup costs 25 times the cheapest
Northwestern University (Wen and Liu, arXiv cs.AI) · Higher education research and research agents · United States; campaigns use published experiments from international labs
- Publisher
- arXiv
- Original publication
- 13 October 2026 (arXiv v1)
- Source retrieved
- 2026-10-19
What happened
Autonomous Science Bench v0.1 gives agents six budgeted design-build-test-learn campaigns (two each in biology, chemistry and materials) built on published experiments, with a lab service that returns measurements and keeps a ledger. A campaign passes only if both a discovery gate and a learning gate pass. In 144 trials across eight agent-model configurations, only 31 passed; the best configuration resolved 67% of its trials and the hardest campaign was solved in three of 24 attempts. Higher success came at steeply higher cost: total cost rose 25-fold from $6.91 to $172.90 between the cheapest and best configurations, and cost per resolved trial from $1.38 to $14.41. Agents failed in different ways: some ranked well but found too few hits, others found hits but could not predict unseen designs.
Why it matters
Research offices and self-driving-lab programmes need ways to judge agent experiment planning before committing instrument time. This benchmark shows success, failure modes and cost vary widely by configuration and campaign.
Evidence and measured results
Opened the arXiv abstract page and the full HTML paper: campaign table, experimental setup, cost and performance results, failure analysis and limitations.
Limitations and uncertainty
Small pilot (six campaigns, eight configurations); replay and digital-twin backends, not live labs; public data may be memorised by models; preprint not peer reviewed; cost figures are API costs for these runs only.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-18; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Labs considering agent-driven experimentation cannot compare agents on reliability and cost before committing instrument time.
- Stakeholders
- Self-driving lab or core facility director
- Research computing director
- Department chairs in chemistry, biology and materials
- Procurement
- Discovery questions
- Do you have past experiments that could be replayed to test an agent?
- How do you compare agent tools on cost per verified outcome?
- Who decides when an agent may run live instruments?
- Value hypothesis
- An evaluation on replayed campus campaigns may avoid committing instrument time and budget to agents that fail on local problems.
- Potential engagement
- A short benchmark study running two or three agent configurations on replayed local campaigns, reporting pass rates and cost per resolved trial.
- Unsupported claims
- Agents reliably run experimental campaigns
- Live-lab performance matches benchmark results
- Fixed cost savings from agent experimentation
Pre-sales engineering
Role takeaway
- Fit
- Fits labs with recorded experimental data and plans for autonomous experimentation; limited to small pilot scope.
- Architecture and integration
- Wrap past experiment data in a budgeted lab service with a ledger and score agents on separate discovery and learning gates.
- Prerequisites
- Recorded experiment datasets with outcomes
- Agent platforms to compare
- Network egress controls during trials
- Constraints
- Replay and twin backends only
- Possible model memorisation of published data
- Six campaigns in three fields
- Security
- Block agent internet access during trials to stop answer retrieval; otherwise not addressed.
- Proof of value
- Run repeated trials per configuration on a local replay campaign
- Report cost per resolved trial
- Check both gates, not overall pass rate
Delivery
Role takeaway
- Work
- Build replay campaigns from local experiments, run agent evaluations, and set a go or no-go gate for live instrument access.
- Dependencies
- Data stewards for past experiments
- Agent platform access
- Budget for API costs
- Ownership
- Core facility or self-driving lab director, with research computing support.
- Skills and adoption
- Staff learn to construct campaigns, budgets and gates, and to read agent ledgers.
- Governance checkpoints
- Gate review before live instrument access
- Cost review per campaign
- Repeat-trial review
- Acceptance criteria
- Proposal: agent passes both gates in a set share of repeated replay trials before live use
- Proposal: cost per resolved trial reported for each configuration
- Proposal: all agent measurement requests recorded in a ledger
- Risks
- Benchmark results not transferring to live labs
- Cost overruns with higher-capability models
- Choosing agents on overall pass rate alone
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
A lab service with an append-only ledger, fixed budgets and separate discovery and learning checks is a usable pattern for evaluating agents before connecting them to instruments.
Governance
Who approves, reviews and stays accountable for outcomes?
Overall success rates hide distinct failure modes, so acceptance tests for agent-run experimentation should check both what was found and what the agent learned.
Security and privacy
What data, permissions and controls need testing?
Not addressed beyond egress controls to stop agents retrieving published answers during trials.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Expert-time estimates for each campaign are provided, but the paper does not measure staff time saved.
Procurement
What should contracts, pricing and exit terms secure?
Cost per resolved trial differed tenfold across configurations; buyers should compare cost per verified outcome, not per run or token.
Operating model
Which teams own the service once it runs?
Agent experimentation should be piloted on replayed or simulated campaigns with budgets before live instrument access.
Publication history
- 2026-10-18Research Universities · Issue 404 resources
Stable resource ID: northwestern-autonomous-science-bench-budgeted-campaigns-20261013