From the Research Universities edition of October 6, 2026
Autonomous Science Bench 0.1: on eight lab-derived pilot campaigns, best agent configuration resolves 70% and most resolve 30% or less; replayed experiments, one trial per most tasks
Autonomous Science Benchmark Team (post by Yibo Wen, Northwestern University), with participating U.S. laboratories; acknowledges NSF support · University research / evaluation of AI agents for autonomous laboratories · United States (tasks from laboratories across the U.S.)
- Publisher
- Autonomous Science Bench project page
- Original publication
- October 1, 2026 (benchmark release page, version 0.1; dateModified 1 Oct)
- Source retrieved
- 2026-10-07
What happened
Version 0.1 scores whole model–agent–harness configurations on eight pilot campaigns from U.S. labs (protein, biosensor and nuclease active learning; propylene and Suzuki chemistry; sparse defect scan and coercivity peak search; inverse lithography). Each campaign gives a seed, a batch budget and fixed rounds; a run resolves only if it passes both a discovery gate and a learning gate, against random, Bayesian-optimization and lab-workflow baselines. As retrieved, Opus 5.5 with Claude Code resolved 70% and GPT-5.6 Terra with Codex 50%; GPT-5.6 Luna and GLM-5.3 Flash 30%; Qwen3.8 27B, Sonnet 5.5 and MiniMax M3 10%; Haiku 4.5 none. No configuration passed the coercivity peak search. Terra cost $43 across the eight against $111 for Opus 5.5. Results come from the v0 replay oracle of recorded experiments; digital-twin and live-lab backends are planned.
Why it matters
Universities building self-driving labs need evaluations tied to real lab outcomes rather than question answering. This release offers a shared framework but shows low and uneven resolution rates, and its results do not yet come from live instruments.
Evidence and measured results
Opened the project page via HTTPS (curl, text and JSON-LD extracted; datePublished and dateModified 1 Oct 2026; author affiliation Northwestern University in page metadata). Read overview, task list, results, scoring and roadmap. The leaderboard figures differed from an earlier search-index snapshot of the page (which showed 63% leaders), so numbers here are as retrieved on 6 Oct.
Limitations and uncertainty
Version 0.1 pilot with eight tasks; one trial on most tasks (three for nuclease); replayed recorded experiments, not live instruments; leadership and contributor names not shown in the retrieved text; page figures appear to have been revised since first indexing; no peer review; published 1 Oct, before the prior run, but not previously archived and included because its results were revised and it is the first lab-campaign benchmark in this stream's archive.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-06; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Labs building autonomous experimentation lack a way to show that an AI agent chooses experiments better than standard optimization before it touches instruments.
- Stakeholders
- Self-driving lab directors
- Research computing director
- Vice president for research
- Instrument core managers
- Discovery questions
- Do you have recorded experiment campaigns that could be replayed to test agents?
- What baseline (random, Bayesian optimization, lab protocol) does the lab currently use?
- Who approves an agent selecting experiments on live instruments?
- Value hypothesis
- Replay-based qualification can screen out agents that do not beat existing optimization before instrument time is spent; the benchmark does not measure savings.
- Potential engagement
- Five-week campaign-replay study on two of a lab's recorded campaigns, comparing three agent configurations with Bayesian optimization over repeated trials.
- Unsupported claims
- Agents reliably run autonomous labs
- Results on live instruments
- Statistically meaningful rankings between top configurations
Pre-sales engineering
Role takeaway
- Fit
- Universities with automated or high-throughput labs in biology, chemistry, materials or semiconductor fabrication that log experiment campaigns.
- Architecture and integration
- Common agent API over three backends (replay oracle, digital twin, live lab); batch-and-refit campaign loop; two-gate grading; baselines run under matched seeds and budgets.
- Prerequisites
- Recorded experiment data with measured outcomes
- Defined campaign budgets and rounds
- Baseline optimizer implementation
- Agent sandbox with cost logging
- Constraints
- Pilot has eight tasks
- One trial on most tasks
- Replay cannot capture instrument faults or queues
- Hosted-model use may conflict with unpublished data terms
- Security
- Keep final evaluation data private from agents, restrict agent access to experiment interfaces only, and require human interlocks before any live-instrument action.
- Proof of value
- Run three or more trials per configuration on local replays
- Compare against Bayesian optimization at matched budget
- Track cost per resolved campaign
- Test time-limit behavior on long campaigns
Delivery
Role takeaway
- Work
- Create a staged qualification path (replay, simulator, supervised live run) for agent-driven experiment selection, with documented gates at each stage.
- Dependencies
- Lab-curated campaign data
- Digital twin or simulator where available
- Instrument safety review
- Model access approvals
- Ownership
- Lab director owns scientific gates and instrument safety; research computing owns runtime and logs; research office owns data agreements.
- Skills and adoption
- Train lab staff to set campaign budgets, read discovery and learning gates, and supervise agent proposals before execution.
- Governance checkpoints
- Replay qualification before simulator stage
- Safety review before live-instrument access
- Human approval of each agent-proposed batch in early live runs
- Re-qualification on model upgrades
- Acceptance criteria
- Proposed: agent beats Bayesian optimization on both gates across at least three trials
- Proposed: cost per campaign recorded and within budget
- Proposed: no live-instrument action without human approval during pilot
- Proposed: failures and time-outs logged per campaign
- Risks
- Over-reading single-trial leaderboard gaps
- Replay results not transferring to live labs
- Agents exhausting time budgets
- Instrument damage or waste in live runs
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
The same agent API is meant to run against a replay oracle, a digital twin and a live lab, which suggests campuses can stage self-driving-lab pilots from recorded data to instruments without changing the agent interface.
Governance
Who approves, reviews and stays accountable for outcomes?
Labs should require campaign-level evidence against Bayesian-optimization and lab baselines before letting an agent choose experiments on real instruments, and treat single-trial leaderboard gaps as noise.
Security and privacy
What data, permissions and controls need testing?
The benchmark uses private final evaluation and controlled data access against contamination; live-lab use would add instrument safety and access-control questions the page does not address.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Labs need staff who can curate campaign data and supervise agents; the benchmark relies on labs' own scientists to judge success.
Procurement
What should contracts, pricing and exit terms secure?
Configuration costs varied from under $6 to $111 across the pilot, so procurement should compare cost per resolved campaign, not model price alone.
Operating model
Which teams own the service once it runs?
Agent-driven experimentation becomes a staged service spanning lab, research computing and safety oversight rather than a tool a single lab adopts alone.
Publication history
- 2026-10-06Research Universities · Issue 295 resources
Stable resource ID: autonomous-science-bench-v01-lab-campaigns-20261001