{"resourceId":"scihorizon-elab-scivlabench-20260925","versions":[{"version":"external-0e5b94862e225418af94b5957e9118a968f99ebe6c5d4358fb4e0a63f154e21b","resource":{"id":"scihorizon-elab-scivlabench-20260925","title":"SciHorizon-eLab/SciVLABench: protocol-to-task compiler; strongest embodied policy SR 49.7%; human-agent coord SR ≤16.7%","organization":"Computer Network Information Center, Chinese Academy of Sciences / Beihang University — arXiv technical report","sector":"Academic AI-for-science / embodied laboratory benchmarking","geography":"China (CAS CNIC / Beihang); transferable methods with limits","publishedAt":"September 25, 2026 (arXiv)","publicationDate":"2026-09-25","eventDate":null,"sourceName":"arXiv 2609.30971","sourceLabel":"Academic preprint (arXiv)","sourceUrl":"https://arxiv.org/abs/2609.30971","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","operating-model","knowledge-work"],"finding":"SciHorizon-eLab is an agentic protocol-to-task compiler that turns natural-language scientific protocols into MuJoCo-certified embodied tasks with independently generated step-level success specs. SciVLABench releases 300 certified tasks (51 scenarios, five operation families) plus 15 human-agent coordination tasks. On ten representative tasks, the strongest visuomotor policy (ACT) averages 49.7% task success; long-horizon Transport and Mix reaches at most 3.3% complete success despite ~40% step-wise progress. Human-agent coordination success stays ≤16.7% despite zero premature violations. A 100-candidate construction audit: 58% first-pass certification, 81% after automatic recovery, 19% needing brief human correction.","sledRelevance":"University programs evaluating scientific embodied agents need sealed, protocol-grounded benchmarks with independent success specs—aggregate demo videos obscure ordered-step and coordination failures.","evidence":"arXiv 2609.30971 opened via HTTPS; 49.7% SR, coordination rates, 300-task scope, and 58%/81%/19% certification audit taken from opened technical report text (dated 25 Sep 2026).","architectureImplications":"Three-stage compilation (semantic grounding, executable+independent eval specs, simulation certification) with separate execution and condition-planning agents; MuJoCo host-side verification.","governanceImplications":"Independent success specifications and simulation certification before benchmark admission are the control layer that prevents execution-from-eval contamination.","securityPrivacyImplications":"Simulation setting lowers physical safety risk but does not address wet-lab biosafety or data governance for real protocols; treat as eval infrastructure.","caveats":"Simulation not physical wet-lab transfer; policy comparison on ten tasks; preprint/technical report; Chinese institutional setting.","streamIds":["research"],"roles":{"sales":"Sales framing: Buyers of scientific embodied agents need hard success rates under independent step-level specs, not cherry-picked demos. SciVLABench’s 49.7% strongest average SR and at most 16.7% human-agent coordination SR support a cautionary value story: protocol-grounded sealed evaluation exposes gaps that launch videos hide. Do not sell wet-lab transfer or guaranteed high-eighty autonomy percentages—those outcomes are not in this technical report and would overclaim the simulation evidence.","engineering":"Engineering framing: Fit is a protocol-to-task compiler with independent evaluation predicates and simulation certification before release. Integrate semantic grounding, skill libraries, and host-side verification separate from the builder agent. Prerequisites include registered skills and predicates plus MuJoCo or equivalent assets. Proof-of-value should report task and step-wise success under ordered specs plus coordination premature-violation and latency metrics—not builder self-scores or uncertified demos.","delivery":"Delivery framing: Stand up compilation and certification pipelines, freeze ordered-step evaluation rubrics, and publish baseline success rates before claiming embodied-lab readiness for campus automation programs. Ownership: platform team owns the compiler and certification host; science leads own protocol semantics; the evaluation harness stays outside the learning agent’s write path. Acceptance should require a certified task package, published baselines, and a ban on agent self-report as the sole success metric. Risks include sim-to-real overclaim, admitting uncertified tasks, and ignoring human-agent coordination failures."},"roleDetails":{"sales":{"customerProblem":"Embodied scientific-agent vendors claim progress without protocol-grounded, independently scored benchmarks.","stakeholders":["VPR / AI-for-science leads","Robotics/core facility directors","Research computing","Research integrity","PI wet-lab automation teams"],"discoveryQuestions":["Are success criteria independent of the builder agent?","What is SR vs SSR under ordered steps?","How are human-agent coordination failures measured?","What certification rate does the task factory achieve?"],"valueHypothesis":"A protocol-to-task compiler with simulation certification can produce scalable sealed evals that reveal sub-50% policy success and coordination gaps before wet-lab spend.","potentialEngagement":"Pilot compiling 10 campus protocols into certified tasks and baselining one visuomotor policy under ordered step metrics.","unsupportedClaims":["Wet-lab transfer success","Average SR above 49.7% for current policies","Zero human certification effort","Clinical or chemistry discovery outcomes"]},"engineering":{"fit":"Agentic compiler producing environment, executable program, and independent success specs with MuJoCo certification.","architectureIntegration":"Separate execution-planning and condition-planning agents; deterministic validators; host-side predicate+visual review before admission.","prerequisites":["Protocol corpus","Atomic skill library","Verification predicates","Simulator assets","Certification audit harness"],"constraints":["Sim-only evidence","Ten-task policy table","Preprint","Asset-library coverage limits"],"security":"Keep eval labels and certification outside the learning agent’s write path; treat real-protocol IP carefully if compiling proprietary SOPs.","proofOfValue":["SR/SSR on certified tasks","Coordination SR/PVR/latency","First-pass and recovered certification rates","Held-out binding generalization gap"]},"delivery":{"work":"Deploy compiler, certify task set, freeze rubrics, run policy baselines, publish gaps.","dependencies":["Simulation cluster","Protocol authors","Robotics eng","Eval reviewers"],"ownership":"Platform owns certification; science owns protocol fidelity; research computing owns eval compute.","skillsAdoption":"Train staff to read ordered-step failures and coordination metrics; train PIs not to equate demo SR with discovery.","governanceCheckpoints":["Rubric freeze","Certification audit sample","Sim-to-real claim review"],"acceptanceCriteria":["Certified task package released","Baseline SR/SSR published","Agent self-report not sole metric"],"risks":["Sim-to-real overclaim","Uncertified task leakage","Ignoring coordination failures","Overfitting to skill library"]}},"retrievedAt":"2026-09-27T22:08:21-05:00","enrichedAt":"2026-09-27T22:13:16-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Low coordination SR shows human-in-the-loop skills remain central; compiler automation still leaves 19% tasks needing brief human correction.","procurementImplications":"Fund sealed eval construction and certification compute separately from robot CapEx; require independent success specs in embodied-lab RFPs.","operatingModelImplications":"Scientific embodied autonomy should be operated as compile–certify–baseline before wet-lab scale-up.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.30971","referenceExcerpt":"the strongest policy attains an average success rate of only 49.7%","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}