{"resourceId":"autonomous-science-bench-v01-lab-campaigns-20261001","versions":[{"version":"external-2ace17f7b1c6639cb900eba88123223ffee05127c6e16afd093bf844deb4f6d1","resource":{"id":"autonomous-science-bench-v01-lab-campaigns-20261001","title":"Autonomous Science Bench 0.1: on eight lab-derived pilot campaigns, best agent configuration resolves 70% and most resolve 30% or less; replayed experiments, one trial per most tasks","organization":"Autonomous Science Benchmark Team (post by Yibo Wen, Northwestern University), with participating U.S. laboratories; acknowledges NSF support","sector":"University research / evaluation of AI agents for autonomous laboratories","geography":"United States (tasks from laboratories across the U.S.)","publishedAt":"October 1, 2026 (benchmark release page, version 0.1; dateModified 1 Oct)","publicationDate":"2026-10-01","eventDate":null,"sourceName":"Autonomous Science Bench project page","sourceLabel":"Research benchmark release (not peer-reviewed)","sourceUrl":"https://yibow.me/autonomous-science-bench","evidenceClass":"independent-research","outcomeClass":"mixed","topics":["developers-agents","infrastructure","operating-model"],"finding":"Version 0.1 scores whole model–agent–harness configurations on eight pilot campaigns from U.S. labs (protein, biosensor and nuclease active learning; propylene and Suzuki chemistry; sparse defect scan and coercivity peak search; inverse lithography). Each campaign gives a seed, a batch budget and fixed rounds; a run resolves only if it passes both a discovery gate and a learning gate, against random, Bayesian-optimization and lab-workflow baselines. As retrieved, Opus 5.5 with Claude Code resolved 70% and GPT-5.6 Terra with Codex 50%; GPT-5.6 Luna and GLM-5.3 Flash 30%; Qwen3.8 27B, Sonnet 5.5 and MiniMax M3 10%; Haiku 4.5 none. No configuration passed the coercivity peak search. Terra cost $43 across the eight against $111 for Opus 5.5. Results come from the v0 replay oracle of recorded experiments; digital-twin and live-lab backends are planned.","sledRelevance":"Universities building self-driving labs need evaluations tied to real lab outcomes rather than question answering. This release offers a shared framework but shows low and uneven resolution rates, and its results do not yet come from live instruments.","evidence":"Opened the project page via HTTPS (curl, text and JSON-LD extracted; datePublished and dateModified 1 Oct 2026; author affiliation Northwestern University in page metadata). Read overview, task list, results, scoring and roadmap. The leaderboard figures differed from an earlier search-index snapshot of the page (which showed 63% leaders), so numbers here are as retrieved on 6 Oct.","architectureImplications":"The same agent API is meant to run against a replay oracle, a digital twin and a live lab, which suggests campuses can stage self-driving-lab pilots from recorded data to instruments without changing the agent interface.","governanceImplications":"Interpretation: labs should require campaign-level evidence against Bayesian-optimization and lab baselines before letting an agent choose experiments on real instruments, and treat single-trial leaderboard gaps as noise.","securityPrivacyImplications":"The benchmark uses private final evaluation and controlled data access against contamination; live-lab use would add instrument safety and access-control questions the page does not address.","caveats":"Version 0.1 pilot with eight tasks; one trial on most tasks (three for nuclease); replayed recorded experiments, not live instruments; leadership and contributor names not shown in the retrieved text; page figures appear to have been revised since first indexing; no peer review; published 1 Oct, before the prior run, but not previously archived and included because its results were revised and it is the first lab-campaign benchmark in this stream's archive.","streamIds":["research"],"roles":{"sales":"Sales framing: institutions investing in self-driving labs need evidence that agents improve experiment selection, not just answer questions. Autonomous Science Bench's first release shows the best configuration resolving 70% of eight replayed lab campaigns and most at 30% or less. A bounded offer is a campaign-replay evaluation using a lab's own recorded experiments before any live-instrument pilot. Do not claim autonomous discovery; results are from replayed data with one trial per most tasks.","engineering":"Engineering framing: stage autonomous-lab agents from recorded data to instruments behind one agent API. Replay recorded experiments first, compare against random and Bayesian-optimization baselines, require both a discovery and a learning gate, and log cost per campaign, which here ranged widely by configuration. Run multiple trials before ranking models, and add time limits, since one leading agent exhausted a two-hour budget without submitting.","delivery":"Delivery framing: run self-driving-lab agents through a staged qualification path owned jointly by the lab and research computing. The lab director owns scientific gates and instrument safety, research computing owns the agent runtime and logging, and the research office owns data-sharing terms. Measure resolution against baselines over repeated trials. Interpretation: the criteria below are proposals; the benchmark reports pilot results only."},"roleDetails":{"sales":{"customerProblem":"Labs building autonomous experimentation lack a way to show that an AI agent chooses experiments better than standard optimization before it touches instruments.","stakeholders":["Self-driving lab directors","Research computing director","Vice president for research","Instrument core managers"],"discoveryQuestions":["Do you have recorded experiment campaigns that could be replayed to test agents?","What baseline (random, Bayesian optimization, lab protocol) does the lab currently use?","Who approves an agent selecting experiments on live instruments?"],"valueHypothesis":"Replay-based qualification can screen out agents that do not beat existing optimization before instrument time is spent; the benchmark does not measure savings.","potentialEngagement":"Five-week campaign-replay study on two of a lab's recorded campaigns, comparing three agent configurations with Bayesian optimization over repeated trials.","unsupportedClaims":["Agents reliably run autonomous labs","Results on live instruments","Statistically meaningful rankings between top configurations"]},"engineering":{"fit":"Universities with automated or high-throughput labs in biology, chemistry, materials or semiconductor fabrication that log experiment campaigns.","architectureIntegration":"Common agent API over three backends (replay oracle, digital twin, live lab); batch-and-refit campaign loop; two-gate grading; baselines run under matched seeds and budgets.","prerequisites":["Recorded experiment data with measured outcomes","Defined campaign budgets and rounds","Baseline optimizer implementation","Agent sandbox with cost logging"],"constraints":["Pilot has eight tasks","One trial on most tasks","Replay cannot capture instrument faults or queues","Hosted-model use may conflict with unpublished data terms"],"security":"Keep final evaluation data private from agents, restrict agent access to experiment interfaces only, and require human interlocks before any live-instrument action.","proofOfValue":["Run three or more trials per configuration on local replays","Compare against Bayesian optimization at matched budget","Track cost per resolved campaign","Test time-limit behavior on long campaigns"]},"delivery":{"work":"Create a staged qualification path (replay, simulator, supervised live run) for agent-driven experiment selection, with documented gates at each stage.","dependencies":["Lab-curated campaign data","Digital twin or simulator where available","Instrument safety review","Model access approvals"],"ownership":"Lab director owns scientific gates and instrument safety; research computing owns runtime and logs; research office owns data agreements.","skillsAdoption":"Train lab staff to set campaign budgets, read discovery and learning gates, and supervise agent proposals before execution.","governanceCheckpoints":["Replay qualification before simulator stage","Safety review before live-instrument access","Human approval of each agent-proposed batch in early live runs","Re-qualification on model upgrades"],"acceptanceCriteria":["Proposed: agent beats Bayesian optimization on both gates across at least three trials","Proposed: cost per campaign recorded and within budget","Proposed: no live-instrument action without human approval during pilot","Proposed: failures and time-outs logged per campaign"],"risks":["Over-reading single-trial leaderboard gaps","Replay results not transferring to live labs","Agents exhausting time budgets","Instrument damage or waste in live runs"]}},"retrievedAt":"2026-10-06T22:03:42-05:00","enrichedAt":"2026-10-06T22:10:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Labs need staff who can curate campaign data and supervise agents; the benchmark relies on labs' own scientists to judge success.","procurementImplications":"Configuration costs varied from under $6 to $111 across the pilot, so procurement should compare cost per resolved campaign, not model price alone.","operatingModelImplications":"Agent-driven experimentation becomes a staged service spanning lab, research computing and safety oversight rather than a tool a single lab adopts alone.","sourceVerification":{"openedUrl":"https://yibow.me/autonomous-science-bench","referenceExcerpt":"With one trial on most tasks, small gaps between configurations are not yet meaningful.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}