From the Research Universities edition of October 2, 2026
Agents Are Systems: ~54% run noise; task information outranks model and budget on four scientific specialist tasks
Technical University of Munich (MCML), Helmholtz Munich, Inria / ENS / CNRS / PSL — arXiv preprint (cs.AI) · Higher education AI-for-science agent evaluation / research computing agent platforms · Germany / France (authors); Jülich/Helmholtz HPC used; methods transferable to campus agent configuration practice
- Publisher
- arXiv 2610.01618
- Original publication
- October 1, 2026 (arXiv 2610.01618v1)
- Source retrieved
- 2026-10-03
What happened
Agents Are Systems, Not Models (arXiv 2610.01618, submitted 1 Oct 2026) evaluates coding agents that must find and operate published specialist models on four scientific tasks (redshift estimation, astronomy MMLU, promoter prediction, RNA folding). Five configuration axes vary: information, reasoning, prompted verification, time budget, and backbone size (8,640 main runs; >18,000 trajectories released). About 54% of outcome variance among genuine attempts is run-to-run noise rather than configuration. Information has the largest effect on gap_closed on every task, exceeding model and budget, and also lowers cost while improving calibration. Prompted self-verification ranks last; adding an oracle verification tool raises scores. Axes interact: more time helps mainly when information or model capacity is already sufficient. Agents almost never decline a weaker specialist on the gap-negative task.
Why it matters
Universities configuring research agents should invest first in task information and system-level verification tools, report multi-run variance, and avoid treating a single success rate or larger model as the primary lever.
Evidence and measured results
arXiv abs HTML and PDF via HTTPS curl; abstract 54% variance and information-axis claims confirmed in pdftotext; check_sources not archived.
Limitations and uncertainty
Preprint; four tasks/two domains; specialists pre-provisioned in sandbox; main grid uses FP8 Qwen3.5 open-weight models.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-02; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Campuses buy larger models for research agents without controlling information, budgets, or verification—and single-run scores mislead.
- Stakeholders
- Research computing directors
- AI platform owners
- Domain PIs
- Research admin / FinOps
- Discovery questions
- How many repeats do we run per agent config?
- What task information is packaged today?
- Do we offer system verification tools or only prompt text?
- Value hypothesis
- Information packs and system verification can raise scientific agent gap_closed more than model upsizing while reducing cost and improving calibration.
- Potential engagement
- Five-week multi-run configuration bake-off on two campus specialist-model tasks with variance/cost dashboards.
- Unsupported claims
- Single-run success rates
- Model size as primary lever
- Prompted self-verification sufficiency
Pre-sales engineering
Role takeaway
- Fit
- Configurable scientific coding agent harness with information/budget/model/verification axes and trajectory release.
- Architecture and integration
- Parameterize system prompt addenda, wall-clock budgets, backbone choice, and optional scoring tools; retain specialist weights/envs in sandboxes.
- Prerequisites
- GPU sandboxes
- Specialist model packages
- Multi-run orchestration
- Cost/token metering
- Constraints
- Four-task evidence base
- Pre-provisioned specialists
- FP8 open-weight main grid
- Preprint
- Security
- Allowlist web fetch; audit oracle use for hill-climbing; isolate specialist credentials per sandbox.
- Proof of value
- Information-axis effect size
- Within-cell variance share
- Cost/calibration deltas
- Oracle vs prompt verification comparison
Delivery
Role takeaway
- Work
- Stand up harness axes, run multi-repeat grids on two workflows, review taxonomy of failures, decide default information/verification packs.
- Dependencies
- Sandbox fleet
- Specialist READMEs/envs
- Eval labels/oracles
- Dashboarding
- Ownership
- Research computing owns harness defaults; PIs own task information content and scientific acceptance.
- Skills and adoption
- Train operators to read multi-run CIs and to prefer system tools over prompt-only verification requests.
- Governance checkpoints
- Axis freeze
- Mid-grid variance review
- Default-config go/no-go
- Acceptance criteria
- ≥3 repeats per cell reported
- Information pack beats model-only baseline on gap_closed
- Cost within band
- Risks
- Run-noise misread as product failure
- Oracle misuse
- Narrow task transfer
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Design agent platforms as configurable systems: information packs, budgets, models, and verification tools as first-class knobs with trajectory logging—not a fixed chat agent.
Governance
Who approves, reviews and stays accountable for outcomes?
Require multi-run reporting (completion, variance, cost, calibration) before accepting agent performance claims for scientific workflows.
Security and privacy
What data, permissions and controls need testing?
Sandboxed GPU agents with web fetch need allowlists; oracle/scoring tools must not become answer-search channels without audit.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Configuration-first agent ops creates roles for research computing staff who package task information and verification tools—not only prompt engineers.
Procurement
What should contracts, pricing and exit terms secure?
RFPs for research agents should require multi-run variance, information-pack options, and system verification—not a single model SKU.
Operating model
Which teams own the service once it runs?
Default agent runbooks should set information and verification tools before escalating model size or wall-clock budget.
Publication history
- 2026-10-02Research Universities · Issue 253 resources
Stable resource ID: agents-are-systems-scientific-evaluation-20261001