{"resourceId":"asi-bench-guidance-withdrawal-autonomy-20260818","versions":[{"version":"external-dbef9f2527b82962097de2411f6c863323b90ff125e4d9b73f601cbffd33e60e","resource":{"id":"asi-bench-guidance-withdrawal-autonomy-20260818","title":"ASI-Bench: average scores fall 50.91→26.62 as methodological guidance is withdrawn","organization":"ASI-Bench consortium (arXiv preprint; multi-institution)","sector":"AI scientific autonomy evaluation","geography":"Multi-institution international authors; transferable caution with task-set limits","publishedAt":"August 18, 2026 (arXiv v1)","publicationDate":"2026-08-18","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint — ASI-Bench project-level autonomy benchmark","sourceUrl":"https://arxiv.org/html/2608.17271","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work","operating-model","governance-procurement"],"finding":"ASI-Bench evaluates innovative exploration and autonomous scientific execution on 60 expert-built project-level tasks across 11 domains, progressively withdrawing methodological guidance. Across 18 state-of-the-art agent–model configurations, average score drops from 50.91 with full methodological guidance to 29.10 with only the method specified and 26.62 when agents must determine the method themselves—showing heavy remaining dependence on human guidance.","sledRelevance":"Why now: Older than last completed run (2026-09-18) but newly relevant against Sep 17 Virtual Biotech-scale multi-agent autonomy narratives and score-only campus messaging; supplies measured guidance-tier collapse still absent from the research-stream archive.","evidence":"60 tasks; 18 configurations; B1/B2/B3 averages 50.91/29.10/26.62; expert-hour construction claim; sandbox/scorer validation narrative.","architectureImplications":"Separate assisted research-agent modes from method-selection autonomy in product architecture and eval harnesses.","governanceImplications":"Ban autonomy marketing unless method-selection (B3-like) scores are disclosed beside guided demos.","securityPrivacyImplications":"Autonomy tests need sandboxed tool egress controls as agents explore methods independently.","caveats":"Aug 18 preprint; expert-constructed tasks may not match campus workloads; not a SLED production ROI study; does not refute guided-agent usefulness.","streamIds":["research"],"roles":{"sales":"Sales framing: Vendors may demo AI research agents under rich methodological guidance and imply end-to-end autonomy. ASI-Bench evidence supports a cautionary discovery conversation: average scores drop from 50.91 with full methodological guidance to 29.10 with only the method named and 26.62 when agents must choose methods—across 18 agent–model configurations on 60 expert-built project tasks. Pitch bounded enablement with explicit guidance levels, not ASI readiness. Unsupported: that campus agents are near autonomous project science.","engineering":"Engineering framing: Fit is a project-level benchmark that progressively withdraws methodological guidance (B1→B3) under sandbox execution and scorer validation. Architecture implication for campuses: separate assisted workflows from autonomy claims and measure both. Prerequisites: task definitions with guidance tiers, sandboxes, and scorers. Constraints: expert-constructed tasks (31,000+ human hours claimed); not a SLED production deployment study; Aug 2026 preprint older than last run—included for why-now against same-week multi-agent deployment narratives. Security: sandbox boundaries for agent tool use. Proof of value is score deltas across guidance tiers, not single leaderboard numbers.","delivery":"Delivery framing: Require any research-agent pilot to report performance at full guidance, method-only, and method-selection tiers before leadership language about autonomy. Dependencies: evaluation owners and compute sandboxes. Ownership: research computing/eval owns tiered scoring; PIs own scientific tasks. Skills adoption trains staff to specify guidance level in every demo. Governance checkpoints: autonomy-claim review against B3 scores. Acceptance: documented tiered scores; risks include demoing only B1 and generalizing."},"roleDetails":{"sales":{"customerProblem":"Autonomy marketing for research agents often reflects guided demos, not method-selection performance.","stakeholders":["VPR","Research computing","Faculty buyers","Communications"],"discoveryQuestions":["At which guidance tier was the demo run?","What is B3/method-selection score?","Who built the evaluation tasks?","What sandbox limits apply?"],"valueHypothesis":"Tiered-guidance evaluation can prevent overclaiming autonomy when average scores fall from ~51 (full guidance) to ~27 (method selection) as in ASI-Bench.","potentialEngagement":"Guidance-tier evaluation add-on for an existing research-agent pilot before autonomy messaging.","unsupportedClaims":["Agents are near ASI","B1 demo equals project autonomy","Guaranteed cross-domain transfer"]},"engineering":{"fit":"Project-level benchmark with progressive withdrawal of methodological guidance and sandbox scoring.","architectureIntegration":"Map campus agent workflows to B1/B2/B3 guidance contracts; instrument sandboxes and scorers accordingly.","prerequisites":["Task specs with guidance tiers","Sandbox execution","Scorer validation","Token/time budgeting"],"constraints":["Expert-built task set","Preprint evaluation","Not institutional ROI study"],"security":"Contain agent tools in sandboxes; control egress during autonomy tests.","proofOfValue":["Score delta B1→B3","Per-configuration variance","Documented guidance contract per run"]},"delivery":{"work":"Define guidance tiers for one pilot, run evaluations at each tier, and gate autonomy language on B3 results.","dependencies":["Eval owners","Sandbox compute","Comms review"],"ownership":"Eval lead owns tier scores; PIs own task content.","skillsAdoption":"Require staff to label every demo with guidance tier.","governanceCheckpoints":["Autonomy-claim review","Sandbox egress check","Score disclosure"],"acceptanceCriteria":["B1/B2/B3 scores recorded","Comms aligned to lowest accepted tier","Limitations published"],"risks":["B1-only demos","Score shopping","Ignoring token/time cost of harder tiers"]}},"retrievedAt":"2026-09-20T03:13:20Z","enrichedAt":"2026-09-20T03:13:20Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Highlights need for method-literate humans when guidance is withdrawn; autonomy gaps imply continued specialist staffing.","procurementImplications":"Demand guidance-tier scorecards in RFPs; treat single guided demos as insufficient evidence of autonomy.","operatingModelImplications":"Operate research agents primarily in guided modes until method-selection metrics justify broader autonomy claims.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2608.17271","referenceExcerpt":"the average score drops from 50.91 with full methodological guidance to 29.10","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}