{"resourceId":"agents-are-systems-scientific-evaluation-20261001","versions":[{"version":"external-d75f29e31e507350f0a263bae857d39e0fc4320558141c5b596528935418393d","resource":{"id":"agents-are-systems-scientific-evaluation-20261001","title":"Agents Are Systems: ~54% run noise; task information outranks model and budget on four scientific specialist tasks","organization":"Technical University of Munich (MCML), Helmholtz Munich, Inria / ENS / CNRS / PSL — arXiv preprint (cs.AI)","sector":"Higher education AI-for-science agent evaluation / research computing agent platforms","geography":"Germany / France (authors); Jülich/Helmholtz HPC used; methods transferable to campus agent configuration practice","publishedAt":"October 1, 2026 (arXiv 2610.01618v1)","publicationDate":"2026-10-01","eventDate":null,"sourceName":"arXiv 2610.01618","sourceLabel":"Academic preprint (arXiv)","sourceUrl":"https://arxiv.org/abs/2610.01618","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","operating-model","infrastructure","governance-procurement"],"finding":"Agents Are Systems, Not Models (arXiv 2610.01618, submitted 1 Oct 2026) evaluates coding agents that must find and operate published specialist models on four scientific tasks (redshift estimation, astronomy MMLU, promoter prediction, RNA folding). Five configuration axes vary: information, reasoning, prompted verification, time budget, and backbone size (8,640 main runs; >18,000 trajectories released). About 54% of outcome variance among genuine attempts is run-to-run noise rather than configuration. Information has the largest effect on gap_closed on every task, exceeding model and budget, and also lowers cost while improving calibration. Prompted self-verification ranks last; adding an oracle verification tool raises scores. Axes interact: more time helps mainly when information or model capacity is already sufficient. Agents almost never decline a weaker specialist on the gap-negative task.","sledRelevance":"Universities configuring research agents should invest first in task information and system-level verification tools, report multi-run variance, and avoid treating a single success rate or larger model as the primary lever.","evidence":"arXiv abs HTML and PDF via HTTPS curl; abstract 54% variance and information-axis claims confirmed in pdftotext; check_sources not archived.","architectureImplications":"Design agent platforms as configurable systems: information packs, budgets, models, and verification tools as first-class knobs with trajectory logging—not a fixed chat agent.","governanceImplications":"Require multi-run reporting (completion, variance, cost, calibration) before accepting agent performance claims for scientific workflows.","securityPrivacyImplications":"Sandboxed GPU agents with web fetch need allowlists; oracle/scoring tools must not become answer-search channels without audit.","caveats":"Preprint; four tasks/two domains; specialists pre-provisioned in sandbox; main grid uses FP8 Qwen3.5 open-weight models.","streamIds":["research"],"roles":{"sales":"Sales framing: agent ROI for scientific workflows is a configuration problem, not a model-size purchase. Agents Are Systems (TUM/Helmholtz/Inria; arXiv 2610.01618, 1 Oct 2026) shows ~54% of outcome variance is run-to-run noise, and task information outranks model and budget on four specialist-model science tasks while cutting cost and improving calibration. Prompted self-check barely moves verification behavior; system tools do. Sell information packs plus verification tooling with multi-run scorecards.","engineering":"Engineering framing: expose information levels, budgets, models, and verification tools as harness axes; log trajectories. Expect high run noise—plan ≥3–5 repeats per cell. Prefer system verification tools over prompt-only self-check. Measure completion, gap_closed, cost, runtime, and calibration together; watch Information×Budget/Model interactions.","delivery":"Delivery framing: pick two campus specialist-model workflows, vary information packs and verification tools under fixed budgets, and report multi-run means with CIs before production. Dependencies include sandboxes, specialist environments, and score oracles owned by research computing. Acceptance needs documented information-axis gains and variance reporting. Risks include overfit to four-task benchmark and oracle hill-climbing."},"roleDetails":{"sales":{"customerProblem":"Campuses buy larger models for research agents without controlling information, budgets, or verification—and single-run scores mislead.","stakeholders":["Research computing directors","AI platform owners","Domain PIs","Research admin / FinOps"],"discoveryQuestions":["How many repeats do we run per agent config?","What task information is packaged today?","Do we offer system verification tools or only prompt text?"],"valueHypothesis":"Information packs and system verification can raise scientific agent gap_closed more than model upsizing while reducing cost and improving calibration.","potentialEngagement":"Five-week multi-run configuration bake-off on two campus specialist-model tasks with variance/cost dashboards.","unsupportedClaims":["Single-run success rates","Model size as primary lever","Prompted self-verification sufficiency"]},"engineering":{"fit":"Configurable scientific coding agent harness with information/budget/model/verification axes and trajectory release.","architectureIntegration":"Parameterize system prompt addenda, wall-clock budgets, backbone choice, and optional scoring tools; retain specialist weights/envs in sandboxes.","prerequisites":["GPU sandboxes","Specialist model packages","Multi-run orchestration","Cost/token metering"],"constraints":["Four-task evidence base","Pre-provisioned specialists","FP8 open-weight main grid","Preprint"],"security":"Allowlist web fetch; audit oracle use for hill-climbing; isolate specialist credentials per sandbox.","proofOfValue":["Information-axis effect size","Within-cell variance share","Cost/calibration deltas","Oracle vs prompt verification comparison"]},"delivery":{"work":"Stand up harness axes, run multi-repeat grids on two workflows, review taxonomy of failures, decide default information/verification packs.","dependencies":["Sandbox fleet","Specialist READMEs/envs","Eval labels/oracles","Dashboarding"],"ownership":"Research computing owns harness defaults; PIs own task information content and scientific acceptance.","skillsAdoption":"Train operators to read multi-run CIs and to prefer system tools over prompt-only verification requests.","governanceCheckpoints":["Axis freeze","Mid-grid variance review","Default-config go/no-go"],"acceptanceCriteria":["≥3 repeats per cell reported","Information pack beats model-only baseline on gap_closed","Cost within band"],"risks":["Run-noise misread as product failure","Oracle misuse","Narrow task transfer"]}},"retrievedAt":"2026-10-02T22:05:20-05:00","enrichedAt":"2026-10-02T22:12:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Configuration-first agent ops creates roles for research computing staff who package task information and verification tools—not only prompt engineers.","procurementImplications":"RFPs for research agents should require multi-run variance, information-pack options, and system verification—not a single model SKU.","operatingModelImplications":"Default agent runbooks should set information and verification tools before escalating model size or wall-clock budget.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.01618","referenceExcerpt":"approximately 54% of the outcome variance coming from repeating the same configuration","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}