From the NVIDIA edition of November 5, 2026
NVIDIA NeMo Compass and Eval Author: turn agent-trace failures into tests and harness fixes; Dassault example claims 28% score gain
NVIDIA · Agent development tooling · Global (Dassault Systèmes engineering example)
- Publisher
- NVIDIA Technical Blog
- Original publication
- 2026-11-04
- Source retrieved
- 2026-11-06
- Event date
- 2026-11-04
What happened
On Nov 4 NVIDIA published a NemoClaw agent-optimization tutorial using NeMo Compass (trace analysis), NeMo Eval Author (Harbor/NeMo Gym evaluations) and harness edits. The workplace-assistant example withholds two held-out tasks from coding agents and reports development passes rising from 3/6 to 6/6 and held-out from 4/6 to 6/6 after a SOUL.md change. A Dassault Systèmes LEO car-door example claims malformed tool calls fell from 142 to zero, a 28% relative evaluation-score improvement and 35% fewer tool calls across six scenarios after an overnight unattended loop. The post itself warns to keep held-out cases hidden to avoid overfitting.
Why it matters
Applicable to campus and agency teams building internal agents (IT helpdesk, research assistants, engineering copilots) who need a failure-to-test loop rather than generic benchmark scores. Dassault results are partner/vendor-reported, not a public-sector evaluation.
Evidence and measured results
Vendor tutorial with small tabulated example results and partner-reported Dassault figures; evaluation artifacts not independently published.
Limitations and uncertainty
Vendor and partner claims; small example task counts; overnight loop results not independently audited.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-06; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Domain agents fail in ways generic benchmarks do not explain, and teams lack a failure-to-test loop.
- Stakeholders
- AI product owner
- Platform engineering
- Risk/compliance for agent tool use
- Discovery questions
- Do you store agent traces today?
- What irreversible tools can the agent call?
- Is there a held-out evaluation set?
- Value hypothesis
- Trace-driven tests may reduce repeated tool-use failures if held-out evaluation is enforced.
- Potential engagement
- Evaluation-suite design for one internal agent.
- Unsupported claims
- 28% score gains in customer environments
- 35% fewer tool calls as a general result
- Safe unattended overnight production changes
Pre-sales engineering
Role takeaway
- Fit
- Teams with MCP/tool-using agents and the ability to store ATIF/JSON traces.
- Architecture and integration
- Compass on trace stores; Eval Author with coding agents; harness files (SOUL.md/ETHOS.md); OpenShell for sandboxing.
- Prerequisites
- Trace logging
- Requirements/ETHOS document
- CI to run Harbor/NeMo Gym tasks
- Constraints
- Vendor tutorial path may need NVIDIA API keys
- Small published example sizes
- Partner metrics not reproducible here
- Security
- Redact or gate sensitive trace fields; block irreversible tools without confirmation.
- Proof of value
- Held-out pass rate improves with development set
- Malformed or policy-violating tool calls fall on local traces
Delivery
Role takeaway
- Work
- Pilot one agent: instrument traces, author tests for top Compass findings, change harness, remeasure.
- Dependencies
- Trace store access
- Coding-agent skills for Eval Author
- Change approval for harness merges
- Ownership
- Agent product owner with platform evaluation support.
- Skills and adoption
- Engineers learn to treat evaluations like regression tests.
- Governance checkpoints
- Approve held-out seal
- Review generated verifiers
- Approve production harness merge
- Acceptance criteria
- Proposal: no harness merge unless held-out and development both improve
- Proposal: unauthorized send_message remains a failing verifier
- Risks
- Overfitting to authored tests
- Trace privacy incidents
- Unattended loops changing production behavior
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Trace store plus Compass analysis, Eval Author tasks/verifiers, OpenShell sandbox, iterative harness changes with held-out gates.
Governance
Who approves, reviews and stays accountable for outcomes?
Accept harness changes only when development and held-out sets both improve; human review of generated evaluations is required.
Security and privacy
What data, permissions and controls need testing?
Production traces may contain sensitive tickets/email/files; Compass analysis needs access controls and retention rules.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Shifts agent improvement from ad-hoc prompt edits toward evaluation engineering skills.
Procurement
What should contracts, pricing and exit terms secure?
Requires NVIDIA API key / build.nvidia.com access for the tutorial path; production may use local models.
Operating model
Which teams own the service once it runs?
Creates a recurring optimize-from-traces loop similar to regression testing for agents.
Publication history
- 2026-11-05NVIDIA · Issue 616 resources
Stable resource ID: nvidia-blog-nemo-compass-eval-author-agent-harness-fixes-20261104