From the NVIDIA edition of October 5, 2026
NVIDIA NeMo Relay tutorial: agent traces show a Hermes tool-layer fix lifted Qwen3 Coder 30B from 19/27 to 22/27 tasks but raised calls, data and time; Sonnet 4.5 unchanged
NVIDIA (with Nous Research) · Vendor technical blog / agent observability tooling · Global
- Publisher
- NVIDIA Technical Blog
- Original publication
- 2026-09-30T16:00:00Z
- Source retrieved
- 2026-10-06
- Event date
- 2026-08-06
What happened
Sep 30, 2026 tutorial (modified Oct 1) showing NeMo Relay 0.8.3 capturing Hermes Agent 0.21.1 runs as ATOF event streams, ATIF trajectories and OpenTelemetry/OpenInference spans viewable in Arize Phoenix. In the Hermes ToolPerf case study (Nous Research), an Aug 6, 2026 rerun compared a pinned baseline with tool-layer fixes across nine tasks, three runs per model per arm, 108 runs. Claude Sonnet 4.5 went from 24/27 (89%) to 23/27 (85%), with similar calls and data but mean duration 16 s to 22 s. Qwen3 Coder 30B rose from 19/27 (70%) to 22/27 (81%), while mean LLM calls rose 3.8 to 4.9, tool calls 2.8 to 3.9, tool-result data 16 KB to 33 KB and duration 27 s to 42 s. One task regressed (3.3 to 9.3 turns) and the hidden-file search stayed at 0-33% on both arms. NVIDIA advises calling a change an improvement only when it is repeatable, warns that live-web runs should not rank models, and says traces can contain prompts, tool arguments, results and file paths.
Why it matters
Campus and agency teams building or buying agents need evidence of how an agent reached its answer, not only whether it passed. The case shows a fix that helped one model and not another, and that traces are records that may contain sensitive data.
Evidence and measured results
Opened the full NVIDIA blog post, including Table 2 and the task-level audit. The case study is a small vendor and partner benchmark (nine tasks, 27 runs per arm per model) without confidence intervals; NVIDIA states the traces are checked into a public results directory, which this run did not inspect.
Limitations and uncertainty
Vendor-authored tutorial; small sample (27 runs per arm per model), two models, nine tasks; no statistical tests; the Sonnet one-run difference is within noise by the post's own reading. Performance depends on harness versions pinned in the post.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-06; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Teams changing agent prompts, tools or models cannot tell whether a change actually helped without repeated, verified runs and traces.
- Stakeholders
- AI platform lead
- Application owners
- Privacy or records officer
- Evaluation or QA lead
- Discovery questions
- How do you decide an agent change is an improvement?
- Do you keep traces of agent runs, and for how long?
- Which tasks have automated success checks?
- Value hypothesis
- A small repeatable evaluation protocol can stop regressions and show trade-offs between success, cost and latency.
- Potential engagement
- Bounded agent change-evaluation pilot on two or three customer tasks using traces and verifiers.
- Unsupported claims
- NeMo Relay improves agent success
- Results generalize beyond the two models tested
- The 108-run study is statistically conclusive
Pre-sales engineering
Role takeaway
- Fit
- Good fit as an OTLP-compatible tracing layer for supported agent harnesses.
- Architecture and integration
- Emit ATOF/ATIF and OpenTelemetry spans to Phoenix or another OTLP backend alongside existing logs.
- Prerequisites
- Supported harness such as Hermes
- Deterministic task verifiers
- Pinned model snapshots
- Trace storage
- Constraints
- Small samples
- Traces contain sensitive content
- Live web search makes runs non-repeatable
- Security
- Redact or restrict traces that include prompts, tool arguments and results.
- Proof of value
- Reproduce paired baseline/candidate runs on two customer tasks
- Confirm trace redaction before export
Delivery
Role takeaway
- Work
- Stand up a repeatable agent change review with paired runs, verifiers and trace review.
- Dependencies
- AI platform team
- Records/privacy staff
- Application owners
- Ownership
- The AI platform team owns evaluation runs and traces; records staff own retention.
- Skills and adoption
- Developers learn to judge changes on verified success first, then cost and latency.
- Governance checkpoints
- Task set approval
- A/B result review
- Trace retention decision
- Acceptance criteria
- Changes ship only with repeatable success gain or equal success at lower cost (proposed)
- Trace retention period set and enforced (proposed)
- Risks
- Over-reading small samples
- Sensitive traces kept indefinitely
- Harness version drift
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Agent pipelines can emit OpenTelemetry spans to existing observability tools; plan storage, retention and redaction for traces that include prompts and tool outputs.
Governance
Who approves, reviews and stays accountable for outcomes?
Use paired, repeated, verifier-checked runs before approving agent configuration changes; the post shows one model regressing slightly while another improved.
Security and privacy
What data, permissions and controls need testing?
Traces may contain prompts, responses, tool arguments, results and file paths; review before sharing, and treat them as records subject to retention and public-records rules where applicable.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Trace review requires analysts who can read OpenTelemetry spans; tutorial material helps, but small teams will need training.
Procurement
What should contracts, pricing and exit terms secure?
Ask agent vendors whether they export standard traces (OpenTelemetry/OpenInference) so evaluation is not locked to one tool.
Operating model
Which teams own the service once it runs?
Agent changes become a release process with evidence, like software changes.
Publication history
- 2026-10-05NVIDIA · Issue 307 resources
Stable resource ID: nvidia-nemo-relay-hermes-harness-trace-evaluation-20260930