{"resourceId":"nvidia-blog-nemo-compass-eval-author-agent-harness-fixes-20261104","versions":[{"version":"external-a3cf20d052ba2889f9a21ce7e45d10baf12a483e43cd597498a6c416be0bca20","resource":{"id":"nvidia-blog-nemo-compass-eval-author-agent-harness-fixes-20261104","title":"NVIDIA NeMo Compass and Eval Author: turn agent-trace failures into tests and harness fixes; Dassault example claims 28% score gain","organization":"NVIDIA","sector":"Agent development tooling","geography":"Global (Dassault Systèmes engineering example)","publishedAt":"2026-11-04","publicationDate":"2026-11-04","eventDate":"2026-11-04","sourceName":"NVIDIA Technical Blog","sourceLabel":"NVIDIA Technical Blog: Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes","sourceUrl":"https://developer.nvidia.com/blog/use-nvidia-nemo-to-turn-agent-failures-into-tests-and-harness-fixes/","evidenceClass":"vendor-claim","outcomeClass":"emerging","topics":["developers-agents","operating-model","knowledge-work"],"finding":"On Nov 4 NVIDIA published a NemoClaw agent-optimization tutorial using NeMo Compass (trace analysis), NeMo Eval Author (Harbor/NeMo Gym evaluations) and harness edits. The workplace-assistant example withholds two held-out tasks from coding agents and reports development passes rising from 3/6 to 6/6 and held-out from 4/6 to 6/6 after a SOUL.md change. A Dassault Systèmes LEO car-door example claims malformed tool calls fell from 142 to zero, a 28% relative evaluation-score improvement and 35% fewer tool calls across six scenarios after an overnight unattended loop. The post itself warns to keep held-out cases hidden to avoid overfitting.","sledRelevance":"Applicable to campus and agency teams building internal agents (IT helpdesk, research assistants, engineering copilots) who need a failure-to-test loop rather than generic benchmark scores. Dassault results are partner/vendor-reported, not a public-sector evaluation.","evidence":"Vendor tutorial with small tabulated example results and partner-reported Dassault figures; evaluation artifacts not independently published.","architectureImplications":"Trace store plus Compass analysis, Eval Author tasks/verifiers, OpenShell sandbox, iterative harness changes with held-out gates.","governanceImplications":"Accept harness changes only when development and held-out sets both improve; human review of generated evaluations is required.","securityPrivacyImplications":"Production traces may contain sensitive tickets/email/files; Compass analysis needs access controls and retention rules.","caveats":"Vendor and partner claims; small example task counts; overnight loop results not independently audited.","streamIds":["nvidia"],"roles":{"sales":"Frame NeMo Compass and Eval Author as an evaluation discipline for domain agents, not as guaranteed accuracy gains. The post's own held-out guidance is the sales-safe message. Treat the Dassault 28% and 35% figures as a partner illustration to discuss measurement design, never as expected customer results.","engineering":"Implement the loop with a real trace corpus, ETHOS/requirements file, Harbor or NeMo Gym tasks, and a sealed held-out set. Review Eval Author-generated tests before trusting them. Start with narrow failure modes (for example unauthorized send) and expand coverage; the post notes production suites often need hundreds to thousands of examples.","delivery":"Stand up tracing and a baseline eval suite before any overnight optimization loop. Require dual improvement on development and held-out tasks for merge. Keep humans reviewing verifier logic that blocks irreversible tool calls. Do not automate production harness changes solely from unattended overnight runs without change control."},"roleDetails":{"sales":{"customerProblem":"Domain agents fail in ways generic benchmarks do not explain, and teams lack a failure-to-test loop.","stakeholders":["AI product owner","Platform engineering","Risk/compliance for agent tool use"],"discoveryQuestions":["Do you store agent traces today?","What irreversible tools can the agent call?","Is there a held-out evaluation set?"],"valueHypothesis":"Trace-driven tests may reduce repeated tool-use failures if held-out evaluation is enforced.","potentialEngagement":"Evaluation-suite design for one internal agent.","unsupportedClaims":["28% score gains in customer environments","35% fewer tool calls as a general result","Safe unattended overnight production changes"]},"engineering":{"fit":"Teams with MCP/tool-using agents and the ability to store ATIF/JSON traces.","architectureIntegration":"Compass on trace stores; Eval Author with coding agents; harness files (SOUL.md/ETHOS.md); OpenShell for sandboxing.","prerequisites":["Trace logging","Requirements/ETHOS document","CI to run Harbor/NeMo Gym tasks"],"constraints":["Vendor tutorial path may need NVIDIA API keys","Small published example sizes","Partner metrics not reproducible here"],"security":"Redact or gate sensitive trace fields; block irreversible tools without confirmation.","proofOfValue":["Held-out pass rate improves with development set","Malformed or policy-violating tool calls fall on local traces"]},"delivery":{"work":"Pilot one agent: instrument traces, author tests for top Compass findings, change harness, remeasure.","dependencies":["Trace store access","Coding-agent skills for Eval Author","Change approval for harness merges"],"ownership":"Agent product owner with platform evaluation support.","skillsAdoption":"Engineers learn to treat evaluations like regression tests.","governanceCheckpoints":["Approve held-out seal","Review generated verifiers","Approve production harness merge"],"acceptanceCriteria":["Proposal: no harness merge unless held-out and development both improve","Proposal: unauthorized send_message remains a failing verifier"],"risks":["Overfitting to authored tests","Trace privacy incidents","Unattended loops changing production behavior"]}},"retrievedAt":"2026-11-06T03:59:54Z","enrichedAt":"2026-11-06T04:05:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Shifts agent improvement from ad-hoc prompt edits toward evaluation engineering skills.","procurementImplications":"Requires NVIDIA API key / build.nvidia.com access for the tutorial path; production may use local models.","operatingModelImplications":"Creates a recurring optimize-from-traces loop similar to regression testing for agents.","sourceVerification":{"openedUrl":"https://developer.nvidia.com/blog/use-nvidia-nemo-to-turn-agent-failures-into-tests-and-harness-fixes/","referenceExcerpt":"Accept changes only when they improve both the test cases and the held-out set.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}