{"resourceId":"omicsbench-long-horizon-bioinformatics-agents-arxiv-2610-21074","versions":[{"version":"external-511f55fefb3c62970fe3ff7d2ae9225ef29144a77b5b2d0e16b84db800ee5dd8","resource":{"id":"omicsbench-long-horizon-bioinformatics-agents-arxiv-2610-21074","title":"OmicsBench preprint: across 12 models and three harnesses, AI agents score 0.43–0.57 on 97 multi-step bioinformatics studies, worse on dependent subtasks","organization":"Gideon Shaked and Siddharth Nagisetty (preprint authors; affiliations not shown in the arXiv rendering)","sector":"Academic and research bioinformatics","geography":"International (benchmark built from published studies; transferable to any research computing setting)","publishedAt":"15 October 2026 (arXiv v1)","publicationDate":"2026-10-15","eventDate":null,"sourceName":"arXiv","sourceLabel":"arXiv:2610.21074, OmicsBench","sourceUrl":"https://arxiv.org/abs/2610.21074","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work"],"finding":"OmicsBench (arXiv, 15 October 2026) poses 97 research-study-scale bioinformatics tasks built from published studies and their datasets, broken into 662 dependent subtasks graded deterministically, with 58 biologically informed graders. Across 12 models on three harnesses, mean scores ranged 0.43–0.57 without in-context feedback and 0.54–0.67 with up to three retries; the slowest agent averaged 3.6 hours per task. Scores fell on later turns and on subtasks depending on earlier answers. Failure-mode clustering found agents applying their own statistical filters instead of specified ones and writing answers from memory. In single-pass mode one model cost 12 times another while scoring 0.01 higher. The authors note gated retries invite reward hacking, the harness effect cannot be separated from the model, and the LLM judge was not checked against human annotators.","sledRelevance":"Research computing groups and core facilities are being asked to support agent-run analyses. This gives a measured view of where agents slip in multi-step analysis, which supports keeping analysts in review and checking intermediate outputs.","evidence":"Opened the arXiv abstract page (v1 submitted 15 October 2026) and the full HTML paper. Read the abstract, benchmark construction, results, cost and efficiency, failure modes and limitations sections.","architectureImplications":"Agent workflows for omics analysis need checkpointed intermediate outputs and deterministic checks at each dependent step, not just a final answer check; harness choice affects results.","governanceImplications":"Supports policies that require analysts to review agent-applied filters and parameters against the protocol before results are reported.","securityPrivacyImplications":"Benchmark uses published datasets; real deployments on human genomic data add data-protection controls not tested here.","caveats":"Single preprint, not peer reviewed; small per-domain samples; only two models ran on more than one harness; gated-mode gains are an upper bound because of possible reward hacking; LLM judge not validated against humans; model names reflect the authors' configurations at the time.","streamIds":["research"],"roles":{"sales":"Lead with supervised analysis, not autonomous agents. OmicsBench shows agents scoring roughly half marks on multi-step bioinformatics studies and slipping on dependent steps. Bioinformatics core directors and research computing leaders need a way to use agents with review built in. A bounded engagement pilots agent-assisted analysis with checkpoints on one pipeline. Interpretation: this is one preprint, so it cannot support claims that agents can replace analysts or cut analysis time.","engineering":"Treat the agent as one step in a checked pipeline. OmicsBench found scores drop on dependent subtasks and that agents substitute their own filters, so pre-sales should design checkpointed intermediate outputs, deterministic validators and retry caps. Cost per correct answer varied up to twelvefold for similar scores, so model choice needs a local benchmark. Interpretation; the preprint tests published datasets, not institutional data or controls.","delivery":"Frame delivery as adding agents to a core facility with review gates. Work would cover selecting one pipeline, defining intermediate checks, setting retry limits and logging agent steps for audit. The bioinformatics core owns it with research computing. Interpretation: acceptance should be measured on error rates caught at checkpoints and analyst review time, since the source reports benchmark scores only."},"roleDetails":{"sales":{"customerProblem":"Bioinformatics cores want to use AI agents for multi-step analyses but cannot trust unsupervised results.","stakeholders":["Bioinformatics core director","Research computing director","Lead PIs in genomics","Research integrity office"],"discoveryQuestions":["Which analysis pipelines are candidates for agent assistance?","How are intermediate outputs checked today?","What error rate would block reporting a result?"],"valueHypothesis":"A supervised pilot may show where agents save analyst time without lowering result quality.","potentialEngagement":"A six- to eight-week supervised pilot on one omics pipeline with checkpoint validators and cost-per-correct-step tracking.","unsupportedClaims":["Agents can replace bioinformaticians","Guaranteed time or cost savings","Results transfer to institutional or clinical data"]},"engineering":{"fit":"Fits cores with standard, well-documented pipelines and answer checks; weak fit for novel exploratory analyses.","architectureIntegration":"Agent harness wrapped around a workflow engine with checkpointed outputs, deterministic validators and step logs.","prerequisites":["Documented reference pipeline and expected outputs","Validator scripts for key intermediate results","Logging of agent actions and parameters"],"constraints":["Benchmark uses published data only","Harness and model effects confounded","Model costs vary widely"],"security":"Keep human genomic data in approved environments and restrict which model endpoints may receive it.","proofOfValue":["Compare agent and analyst outputs on ten past analyses with known results","Measure how many agent deviations checkpoints catch","Track cost per correct subtask across two candidate models"]},"delivery":{"work":"Select a pipeline, write checkpoint validators, set retry caps and escalation, and log agent steps for audit.","dependencies":["Analyst time to build validators","Approved model endpoints","Data access approvals"],"ownership":"Bioinformatics core with research computing.","skillsAdoption":"Training analysts to review agent logs and parameters, and to recognise substituted filters.","governanceCheckpoints":["Pipeline and data approval before pilot","Mid-pilot error review","Sign-off before any agent-assisted result is reported"],"acceptanceCriteria":["Proposal: every agent-applied filter matches the protocol or is flagged","Proposal: agent error rate on validated subtasks below an agreed threshold","Proposal: analyst review time per analysis measured against baseline"],"risks":["Silent substitution of analysis parameters","Reward hacking through retries","Overreliance on benchmark scores"]}},"retrievedAt":"2026-10-19T21:58:30-05:00","enrichedAt":"2026-10-19T21:58:30-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Points to continued need for bioinformaticians who can audit agent steps; agents do not remove the review role.","procurementImplications":"Cost varied widely for similar scores, so institutions buying agent services should compare cost per correct subtask, not list price or headline accuracy.","operatingModelImplications":"Core facilities could treat agents as assistants whose intermediate outputs are reviewed, with retry limits and escalation to a human analyst.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.21074","referenceExcerpt":"no combination comes near saturating the benchmark, with mean scores of 0.43 to 0.57 out of a possible 1 without in-context feedback","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}