{"resourceId":"nvidia-nemo-relay-hermes-harness-trace-evaluation-20260930","versions":[{"version":"external-9c4f4e6b9a955741747d801ebcb59d3ef2eba0b17573e85378359fac1deb32af","resource":{"id":"nvidia-nemo-relay-hermes-harness-trace-evaluation-20260930","title":"NVIDIA NeMo Relay tutorial: agent traces show a Hermes tool-layer fix lifted Qwen3 Coder 30B from 19/27 to 22/27 tasks but raised calls, data and time; Sonnet 4.5 unchanged","organization":"NVIDIA (with Nous Research)","sector":"Vendor technical blog / agent observability tooling","geography":"Global","publishedAt":"2026-09-30T16:00:00Z","publicationDate":"2026-09-30","eventDate":"2026-08-06","sourceName":"NVIDIA Technical Blog","sourceLabel":"Vendor tutorial with A/B case study / NeMo Relay","sourceUrl":"https://developer.nvidia.com/blog/tracing-agent-harness-behavior-with-nvidia-nemo-relay/","evidenceClass":"vendor-claim","outcomeClass":"mixed","topics":["developers-agents","data-security","governance-procurement"],"finding":"Sep 30, 2026 tutorial (modified Oct 1) showing NeMo Relay 0.8.3 capturing Hermes Agent 0.21.1 runs as ATOF event streams, ATIF trajectories and OpenTelemetry/OpenInference spans viewable in Arize Phoenix. In the Hermes ToolPerf case study (Nous Research), an Aug 6, 2026 rerun compared a pinned baseline with tool-layer fixes across nine tasks, three runs per model per arm, 108 runs. Claude Sonnet 4.5 went from 24/27 (89%) to 23/27 (85%), with similar calls and data but mean duration 16 s to 22 s. Qwen3 Coder 30B rose from 19/27 (70%) to 22/27 (81%), while mean LLM calls rose 3.8 to 4.9, tool calls 2.8 to 3.9, tool-result data 16 KB to 33 KB and duration 27 s to 42 s. One task regressed (3.3 to 9.3 turns) and the hidden-file search stayed at 0-33% on both arms. NVIDIA advises calling a change an improvement only when it is repeatable, warns that live-web runs should not rank models, and says traces can contain prompts, tool arguments, results and file paths.","sledRelevance":"Campus and agency teams building or buying agents need evidence of how an agent reached its answer, not only whether it passed. The case shows a fix that helped one model and not another, and that traces are records that may contain sensitive data.","evidence":"Opened the full NVIDIA blog post, including Table 2 and the task-level audit. The case study is a small vendor and partner benchmark (nine tasks, 27 runs per arm per model) without confidence intervals; NVIDIA states the traces are checked into a public results directory, which this run did not inspect.","architectureImplications":"Agent pipelines can emit OpenTelemetry spans to existing observability tools; plan storage, retention and redaction for traces that include prompts and tool outputs.","governanceImplications":"Use paired, repeated, verifier-checked runs before approving agent configuration changes; the post shows one model regressing slightly while another improved.","securityPrivacyImplications":"Traces may contain prompts, responses, tool arguments, results and file paths; review before sharing, and treat them as records subject to retention and public-records rules where applicable.","caveats":"Vendor-authored tutorial; small sample (27 runs per arm per model), two models, nine tasks; no statistical tests; the Sonnet one-run difference is within noise by the post's own reading. Performance depends on harness versions pinned in the post.","streamIds":["nvidia"],"roles":{"sales":"Lead with evidence of agent behavior: NVIDIA's Sep 30 NeMo Relay tutorial shows that a Hermes tool fix lifted Qwen3 Coder 30B from 19 to 22 of 27 tasks while nearly doubling tool-result data and raising run time from 27 to 42 seconds, and did nothing for Claude Sonnet 4.5. That is a useful script for agencies that ask whether an agent change is better. Offer a small, repeatable agent evaluation. Do not present this as proof NeMo Relay improves agents; it is a measurement tool, and the sample is small.","engineering":"NeMo Relay fits as an observability layer that emits ATOF, ATIF and OpenTelemetry spans into Phoenix, LangSmith or another OTLP backend. Prerequisites are a supported harness (Hermes has native integration), a deterministic verifier per task and pinned model snapshots. Constraints are trace storage, redaction of prompts and tool outputs, and small samples. Proof of value: rerun the post's protocol on two of the customer's own tasks, with baseline and change, three or more repetitions each, and compare verified success first, then calls, data and time.","delivery":"Delivery work is a repeatable agent change review: fixed tasks with automated checks, paired baseline and candidate runs, and trace review before release. The AI platform team owns the harness and traces; records or privacy staff set retention for traces that contain prompts and results. Checkpoints are the task set approval, the A/B result review and the trace retention decision. Proposed acceptance: a change ships only with a repeatable success gain or equal success at lower cost. Risks include over-reading small samples and storing sensitive traces indefinitely."},"roleDetails":{"sales":{"customerProblem":"Teams changing agent prompts, tools or models cannot tell whether a change actually helped without repeated, verified runs and traces.","stakeholders":["AI platform lead","Application owners","Privacy or records officer","Evaluation or QA lead"],"discoveryQuestions":["How do you decide an agent change is an improvement?","Do you keep traces of agent runs, and for how long?","Which tasks have automated success checks?"],"valueHypothesis":"A small repeatable evaluation protocol can stop regressions and show trade-offs between success, cost and latency.","potentialEngagement":"Bounded agent change-evaluation pilot on two or three customer tasks using traces and verifiers.","unsupportedClaims":["NeMo Relay improves agent success","Results generalize beyond the two models tested","The 108-run study is statistically conclusive"]},"engineering":{"fit":"Good fit as an OTLP-compatible tracing layer for supported agent harnesses.","architectureIntegration":"Emit ATOF/ATIF and OpenTelemetry spans to Phoenix or another OTLP backend alongside existing logs.","prerequisites":["Supported harness such as Hermes","Deterministic task verifiers","Pinned model snapshots","Trace storage"],"constraints":["Small samples","Traces contain sensitive content","Live web search makes runs non-repeatable"],"security":"Redact or restrict traces that include prompts, tool arguments and results.","proofOfValue":["Reproduce paired baseline/candidate runs on two customer tasks","Confirm trace redaction before export"]},"delivery":{"work":"Stand up a repeatable agent change review with paired runs, verifiers and trace review.","dependencies":["AI platform team","Records/privacy staff","Application owners"],"ownership":"The AI platform team owns evaluation runs and traces; records staff own retention.","skillsAdoption":"Developers learn to judge changes on verified success first, then cost and latency.","governanceCheckpoints":["Task set approval","A/B result review","Trace retention decision"],"acceptanceCriteria":["Changes ship only with repeatable success gain or equal success at lower cost (proposed)","Trace retention period set and enforced (proposed)"],"risks":["Over-reading small samples","Sensitive traces kept indefinitely","Harness version drift"]}},"retrievedAt":"2026-10-06T02:57:17Z","enrichedAt":"2026-10-06T03:03:31Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Trace review requires analysts who can read OpenTelemetry spans; tutorial material helps, but small teams will need training.","procurementImplications":"Ask agent vendors whether they export standard traces (OpenTelemetry/OpenInference) so evaluation is not locked to one tool.","operatingModelImplications":"Agent changes become a release process with evidence, like software changes.","sourceVerification":{"openedUrl":"https://developer.nvidia.com/blog/tracing-agent-harness-behavior-with-nvidia-nemo-relay/","referenceExcerpt":"The fixes brought success to tasks the baseline abandoned, and the price was a slower, chattier agent.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}