{"resourceId":"arxiv-2607-12227-rethinking-harness-evolution-evaluation-20261005","versions":[{"version":"external-a3cf20d052ba2889f9a21ce7e45d10baf12a483e43cd597498a6c416be0bca20","resource":{"id":"arxiv-2607-12227-rethinking-harness-evolution-evaluation-20261005","title":"Preprint: automatic harness evolution often fails matched-budget baselines and held-out generalization on Terminal-Bench","organization":"arXiv (Wang et al.)","sector":"Academic AI agent evaluation research","geography":"Global (benchmark study)","publishedAt":"2026-10-05","publicationDate":"2026-10-05","eventDate":"2026-10-05","sourceName":"arXiv","sourceLabel":"arXiv:2607.12227 Rethinking the Evaluation of Harness Evolution for Agents (v4)","sourceUrl":"https://arxiv.org/abs/2607.12227","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","knowledge-work"],"finding":"Preprint, not peer reviewed (submitted 14 Jul 2026, last revised 5 Oct 2026, v4). The authors argue harness-evolution evaluations often lack matched-budget test-time-scaling baselines and reuse the same verification signals for search and reported scores. On Terminal-Bench 2.1, automatic harness evolution fails to outperform simple test-time scaling with or without test cases and shows limited generalization to held-out tasks. They find stronger gains in long-horizon games (about 80% over search baseline on ARC-AGI-3; about 11% on EdgeBench) under matched budgets. Newly relevant as independent scrutiny of the Nov 4 NVIDIA NeMo harness-optimization narrative.","sledRelevance":"Transferable caution for public-sector and university agent programs adopting automatic harness optimization: require matched baselines and sealed held-out tasks before accepting score gains. Benchmarks are not U.S. government deployments.","evidence":"arXiv preprint abstract and metadata inspected; full HTML retrieved. Not peer reviewed.","architectureImplications":"Evaluation harnesses need train/test separation and matched inference budgets when comparing automatic evolution to sampling baselines.","governanceImplications":"Procurement and acceptance tests for agent tooling should demand held-out generalization evidence, not search-set scores alone.","securityPrivacyImplications":"Not the paper's focus; evaluation design still affects whether unsafe tool-use regressions are caught.","caveats":"Preprint; Terminal-Bench and game domains may not match campus IT agents; quantitative game gains are setting-specific.","streamIds":["nvidia"],"roles":{"sales":"Use this preprint as a caution beside NVIDIA's Nov 4 NeMo post, not as a dismissal of evaluation tooling. It says automatic harness evolution often fails matched-budget baselines and held-out generalization on Terminal-Bench. For SLED buyers, the ask is simple: show held-out results and a baseline that spends similar inference budget. Do not over-claim peer review; label it preprint.","engineering":"When piloting Compass/Eval Author-style loops, build matched baselines (repeated sampling or sequential refinement) and keep a sealed held-out set. Do not report search-set scores as production readiness. Expect limited transfer from Terminal-Bench findings to domain agents; still adopt the protocol lessons.","delivery":"Add evaluation-design checkpoints to agent projects: document baseline budget, held-out seal, and acceptance rules before overnight optimization. If a vendor cites large relative gains, require the same protocol. Record that this source is a preprint when briefing sponsors."},"roleDetails":{"sales":{"customerProblem":"Vendors report harness-optimization gains without matched baselines or held-out tests.","stakeholders":["AI program sponsor","Evaluation lead","Procurement"],"discoveryQuestions":["What is the held-out set?","What inference budget does the baseline get?","Is the paper or claim peer reviewed?"],"valueHypothesis":"Stronger evaluation protocols reduce false confidence in agent improvements.","potentialEngagement":"Evaluation-protocol review for an agent pilot.","unsupportedClaims":["Peer-reviewed status","Direct transfer of Terminal-Bench results to SLED agents","That all harness optimization is ineffective"]},"engineering":{"fit":"Teams comparing automatic harness search to simpler test-time scaling.","architectureIntegration":"Held-out task split; matched-budget baseline runners; logging of feedback queries.","prerequisites":["Clear task verifiers","Budget accounting for inference/feedback"],"constraints":["Preprint","Benchmark domains differ from campus agents"],"security":"Ensure verifiers catch unsafe tool use, not only task success.","proofOfValue":["Harness changes beat matched baseline on held-out tasks","No reliance on search-set scores alone"]},"delivery":{"work":"Insert matched-baseline and held-out checks into the agent optimization runbook.","dependencies":["Evaluation dataset ownership","Compute budget for baselines"],"ownership":"Evaluation lead with agent engineering support.","skillsAdoption":"Teams learn to separate search feedback from final reporting.","governanceCheckpoints":["Approve held-out seal","Approve baseline budget parity","Label preprint evidence in briefings"],"acceptanceCriteria":["Proposal: report held-out delta vs matched baseline before production harness change","Proposal: preprint caveats included in sponsor readout"],"risks":["Overgeneralizing game-domain gains","Ignoring vendor tools because of one preprint"]}},"retrievedAt":"2026-11-06T03:59:54Z","enrichedAt":"2026-11-06T04:05:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Raises the bar for evaluation literacy among agent builders.","procurementImplications":"Treat vendor harness-optimization claims as needing independent evaluation design review.","operatingModelImplications":"Supports keeping humans in the loop on what counts as improvement.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2607.12227","referenceExcerpt":"automatic harness evolution fails to outperform simple test-time scaling methods both with and without test cases","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}