Lighthouse AdvisorySLED AI Adoption Intelligence
← Back to results

From the NVIDIA edition of November 5, 2026

Academic researchCautionaryPreprint, not peer reviewedRecent

Preprint: automatic harness evolution often fails matched-budget baselines and held-out generalization on Terminal-Bench

arXiv (Wang et al.) · Academic AI agent evaluation research · Global (benchmark study)

NVIDIA
Publisher
arXiv
Original publication
2026-10-05
Source retrieved
2026-11-06
Event date
2026-10-05
Read original source

What happened

Preprint, not peer reviewed (submitted 14 Jul 2026, last revised 5 Oct 2026, v4). The authors argue harness-evolution evaluations often lack matched-budget test-time-scaling baselines and reuse the same verification signals for search and reported scores. On Terminal-Bench 2.1, automatic harness evolution fails to outperform simple test-time scaling with or without test cases and shows limited generalization to held-out tasks. They find stronger gains in long-horizon games (about 80% over search baseline on ARC-AGI-3; about 11% on EdgeBench) under matched budgets. Newly relevant as independent scrutiny of the Nov 4 NVIDIA NeMo harness-optimization narrative.

Why it matters

Transferable caution for public-sector and university agent programs adopting automatic harness optimization: require matched baselines and sealed held-out tasks before accepting score gains. Benchmarks are not U.S. government deployments.

Evidence and measured results

arXiv preprint abstract and metadata inspected; full HTML retrieved. Not peer reviewed.

Limitations and uncertainty

Preprint; Terminal-Bench and game domains may not match campus IT agents; quantitative game gains are setting-specific.

Cross-source patterns this evidence supports

Put this evidence to work

Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-06; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.

Sales

Role takeaway
Customer problem
Vendors report harness-optimization gains without matched baselines or held-out tests.
Stakeholders
  • AI program sponsor
  • Evaluation lead
  • Procurement
Discovery questions
  • What is the held-out set?
  • What inference budget does the baseline get?
  • Is the paper or claim peer reviewed?
Value hypothesis
Stronger evaluation protocols reduce false confidence in agent improvements.
Potential engagement
Evaluation-protocol review for an agent pilot.
Unsupported claims
  • Peer-reviewed status
  • Direct transfer of Terminal-Bench results to SLED agents
  • That all harness optimization is ineffective

Pre-sales engineering

Role takeaway
Fit
Teams comparing automatic harness search to simpler test-time scaling.
Architecture and integration
Held-out task split; matched-budget baseline runners; logging of feedback queries.
Prerequisites
  • Clear task verifiers
  • Budget accounting for inference/feedback
Constraints
  • Preprint
  • Benchmark domains differ from campus agents
Security
Ensure verifiers catch unsafe tool use, not only task success.
Proof of value
  • Harness changes beat matched baseline on held-out tasks
  • No reliance on search-set scores alone

Delivery

Role takeaway
Work
Insert matched-baseline and held-out checks into the agent optimization runbook.
Dependencies
  • Evaluation dataset ownership
  • Compute budget for baselines
Ownership
Evaluation lead with agent engineering support.
Skills and adoption
Teams learn to separate search feedback from final reporting.
Governance checkpoints
  • Approve held-out seal
  • Approve baseline budget parity
  • Label preprint evidence in briefings
Acceptance criteria
  • Proposal: report held-out delta vs matched baseline before production harness change
  • Proposal: preprint caveats included in sponsor readout
Risks
  • Overgeneralizing game-domain gains
  • Ignoring vendor tools because of one preprint

Implementation considerations

Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.

Architecture and integration

What must connect, and where does the AI sit in the workflow?

Evaluation harnesses need train/test separation and matched inference budgets when comparing automatic evolution to sampling baselines.

Governance

Who approves, reviews and stays accountable for outcomes?

Procurement and acceptance tests for agent tooling should demand held-out generalization evidence, not search-set scores alone.

Security and privacy

What data, permissions and controls need testing?

Not the paper's focus; evaluation design still affects whether unsafe tool-use regressions are caught.

Accessibility and workforce

Who is affected, and what skills or accommodations follow?

Raises the bar for evaluation literacy among agent builders.

Procurement

What should contracts, pricing and exit terms secure?

Treat vendor harness-optimization claims as needing independent evaluation design review.

Operating model

Which teams own the service once it runs?

Supports keeping humans in the loop on what counts as improvement.

Publication history

  1. 2026-11-05NVIDIA · Issue 616 resources
Read preserved resource versions (JSON)

Stable resource ID: arxiv-2607-12227-rethinking-harness-evolution-evaluation-20261005