{"resourceId":"labagent-lab-skill-harness-life-science-20260911","versions":[{"version":"external-a54edf6d5f648b68aa6cfc7cc28e23bb9ce270f499c6d1a3085a8b179e6eb1c2","resource":{"id":"labagent-lab-skill-harness-life-science-20260911","title":"LabAgent verified lab skills beat Claude Code/Codex on many entries but clear only 2/10 GeneBench-Pro problems","organization":"Yale University and collaborators (arXiv preprint)","sector":"University research / computational biology agent harnesses","geography":"United States (author affiliations); life-science benchmark tasks","publishedAt":"September 11, 2026 (arXiv v1)","publicationDate":"2026-09-11","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint — LabAgent reproduce-and-discovery harness","sourceUrl":"https://arxiv.org/html/2609.13437","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","knowledge-work","operating-model","infrastructure"],"finding":"LabAgent converts a laboratory's papers, code, tutorials, and issues into verified executable skills with corrective-experience memory. Against Claude Code (Opus 4.8) and Codex CLI (GPT-5.5), it landed inside published standard deviations on 32 of 65 entries, achieved best mean rank 2.62, and ranked first on 13/65 versus 8 for each commercial harness, while GeneBench-Pro mean partial score was only 31.4 with 2 of 10 binary verdicts.","sledRelevance":"Why now: 2026-09-17 edition archived Paper2Agent interactive papers and ECAS/SciRIGOR agent-validation limits; LabAgent (11 Sep, still unarchived) adds lab-continuity implementation evidence with measured commercial-harness comparisons and an explicit hard-problem failure mode useful for research-computing delivery.","evidence":"Author comparisons on drug property, biomedical, protein variant, and statistical genetics tasks; figure-reproduction claim; GeneBench-Pro per-problem table with mean 31.4 and 2/10 verdicts.","architectureImplications":"Prefer lab-scoped skill libraries with execute-and-verify locks over unconstrained generalist coding agents for method continuity.","governanceImplications":"Require documented skill verification and human lock approval before treating agent outputs as reusable lab methods.","securityPrivacyImplications":"Lab code and biomedical artifacts in skill corpora need access control and provenance logging.","caveats":"Author-evaluated preprint; commercial CLI versions author-pinned; GeneBench-Pro shows large remaining failure modes; not a multi-campus deployment study.","streamIds":["research"],"roles":{"sales":"Sales framing: University labs lose method continuity when students graduate, and generalist coding agents still miss published error bars on many life-science reproductions. LabAgent evidence supports a bounded pitch around packaging a lab's own papers/code/issues into verified executable skills with corrective memory—not a claim that agents replace PIs or clear GeneBench-class problems. Discovery should map which repositories and figures must reproduce before any 'AI co-scientist' language. Value is continuity and rank gains versus Claude Code/Codex on the authors' 65-entry panel, with explicit unsupported claims around GeneBench-Pro (only 2/10 verdicts).","engineering":"Engineering framing: Fit is a lab-scoped skill library with execute-and-verify loops and persistent corrective experience, integrated beside existing Claude Code/Codex-style CLIs rather than replacing HPC schedulers. Prerequisites include accessible paper/code/tutorial/issue corpora, pinned environments for skill tests, and human review of locked skills. Constraints: author-chosen commercial baselines, life-science task mix, and GeneBench-Pro failures. Security must treat lab code and biomedical data under campus research-computing policy. Proof of value is agreement with published error bars, figure reproduction checks, and documented skill verification—not demo chat.","delivery":"Delivery framing: Pilot one lab's public artifacts into a verified skill library, shadow generalist agents on a fixed reproduction panel, and gate discovery prompts until skills pass execute-and-verify. Dependencies: research computing, lab PI ownership, and IRB/data agreements where human or clinical-adjacent data appear. Ownership stays with the lab; platform teams supply harness operations. Skills adoption teaches staff to record failures as corrective memory. Governance checkpoints include skill lock reviews and GeneBench-style hard-problem sampling. Acceptance: measurable entry-in-error-bar rate and at least one figure reproduction; risks include overclaiming discovery and silent skill drift."},"roleDetails":{"sales":{"customerProblem":"Labs lose the ability to rerun and extend methods as students leave, while generalist agents fail many published reproductions.","stakeholders":["PI / lab lead","Research computing","Graduate researchers","Core facility managers"],"discoveryQuestions":["Which papers/figures must reproduce first?","What code and issue history exist?","How do you compare against Claude Code/Codex today?","Which GeneBench-class problems are in scope?"],"valueHypothesis":"A lab-scoped verified skill harness can improve reproduction rank versus generalist CLIs if skills are locked and failures retained—without claiming broad scientific problem solving.","potentialEngagement":"Single-lab skill-library pilot with a fixed reproduction panel and figure-check acceptance.","unsupportedClaims":["Agents replace human scientific judgment","Guaranteed GeneBench-Pro success","Cross-domain transfer without revalidation"]},"engineering":{"fit":"Lab-specific MCP/skill harness with execute-verify loops and corrective memory alongside existing coding agents.","architectureIntegration":"Ingest papers/repos/tutorials/issues; emit locked skills callable from research notebooks/CLIs; keep human approval before discovery expansion.","prerequisites":["Accessible lab artifact corpus","Pinned runtime for skill tests","Comparison harness vs baseline CLIs","Figure/result oracles"],"constraints":["Author-evaluated baselines","Life-science focus","Hard multi-stage genetics failures (GeneBench-Pro 2/10)"],"security":"Control access to lab code and any regulated biomedical data; audit skill provenance and lock changes.","proofOfValue":["Entries inside published error bars","Figure reproduction match","Mean rank vs Claude Code/Codex","Documented corrective-memory reuse"]},"delivery":{"work":"Stand up artifact ingest, skill verification, baseline comparison panel, and gated discovery prompts for one lab.","dependencies":["PI sponsorship","Research computing runtimes","Data-use agreements if needed"],"ownership":"Lab PI owns scientific acceptance; research computing owns harness operations.","skillsAdoption":"Train researchers to treat skills as verified procedures and to log failures as reusable corrections.","governanceCheckpoints":["Skill lock review","Hard-problem sampling","Baseline re-run after model/tool upgrades"],"acceptanceCriteria":["Agreed in-error-bar rate on panel","One reproduced published figure","Written non-goals for unsolved GeneBench-class tasks"],"risks":["Overclaiming discovery","Silent skill drift","Vendor CLI churn invalidating baselines","Leak of unpublished methods"]}},"retrievedAt":"2026-09-19T03:12:20Z","enrichedAt":"2026-09-19T03:12:20Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Targets continuity when students graduate; still requires skilled staff to curate oracles and interpret GeneBench-class failures.","procurementImplications":"Buy or build lab-scoped harnesses with verification gates; do not procure generalist CLIs alone as method-continuity solutions.","operatingModelImplications":"Shifts some reproduction labor into maintained skill libraries while keeping PIs accountable for scientific acceptance.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2609.13437","referenceExcerpt":"LabAgent landed inside the published standard deviation on 32 of the 65 entries","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}