From the Dell edition of November 7, 2026
Dell InfoHub field note: in the Dell/Broadcom Pikes Peak AI Lab, Metrum AI Ops agents correlated NIC, PowerSwitch and topology telemetry to localize a silent RoCE receiver back-pressure "gray failure" in about five minutes, with remediation staged for human approval
Dell Technologies, Broadcom and Metrum AI (Pikes Peak AI Lab, a Dell Technologies and Broadcom facility) · AI infrastructure operations (vendor validation lab); applicable to university and research-computing GPU clusters · Pikes Peak AI Lab (Dell/Broadcom facility; location not stated in the document); U.S. vendor lab, not a customer deployment
- Publisher
- Dell Technologies InfoHub
- Original publication
- 2026-11-03
- Source retrieved
- 2026-11-08
What happened
A Nov. 3, 2026 Dell InfoHub field note describes a 23-host, 166-GPU lab (PowerEdge XE9680 with AMD MI300X and NVIDIA H200, PowerSwitch Z9864F-ON on Dell SONiC 4.4.0, Broadcom Thor 2 RoCE NICs, PowerScale F910) where switch queue drops were zero and dashboards raised no actionable alert, yet NIC counters showed ~7.3M RNR NAK retries and ~4.48% of packets resent. Metrum AI Ops agents correlated NIC, switch and LLDP topology data, ruled out look-alikes and produced a ranked recommendation in ~5 minutes; remediation was staged for human approval and not applied.
Why it matters
University research-computing centers and state-funded AI platforms increasingly run multi-host GPU training over RoCE Ethernet fabrics with small operations teams. This note shows a failure mode standard dashboards miss and a read-only, human-approved agent workflow for diagnosing it. Relevance is engineering and operations; it is a vendor lab, not a public-sector deployment, and no SLED customer is named.
Evidence and measured results
Vendor-authored single-incident lab investigation. Reported measurements: ~7,308,289 RNR NAK retries and ~7,305,209 receive-side discards on one host; ~4.48% retransmission share; MTTD ~45–90 s, MTTK +~30–90 s, ~5 min end-to-end (marked "Measured"). The document states counter sampling window, reset history and PFC direction were not established, and that it is not a benchmark or guarantee.
Limitations and uncertainty
Single lab incident in a Dell/Broadcom facility using a partner platform (Metrum AI Ops); vendor-authored and promotional framing ("practical validation of the Dell AI Factory operating model"). No independent replication, no workload slowdown magnitude, no duration of the fault, and remediation not applied. The InfoHub page is dated Nov. 3, 2026 (PDF metadata shows creation Sept. 25, 2026). Published before the last completed run; included because it was not archived and adds operational-constraint evidence not covered in prior Dell editions.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-08; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Research computing teams running multi-host GPU training over RoCE fabrics can lose GPU time to faults that leave links up, drops at zero and dashboards quiet, as this Dell lab incident shows.
- Stakeholders
- University research computing director
- HPC/AI cluster network engineers
- CIO or CTO funding AI infrastructure
- Principal investigators reliant on shared GPU clusters
- Discovery questions
- Do you currently collect RNR NAK and PFC pause counters from NICs and switches?
- Can you map each host NIC to its switch port automatically?
- How do you detect training jobs slowed by fabric back-pressure today?
- What approval process governs configuration changes on the cluster fabric?
- Would you allow read-only agent access to iDRAC and switch telemetry?
- Value hypothesis
- Cross-layer telemetry correlation may convert undetected fabric slowdowns into diagnosed incidents within minutes, protecting GPU utilization on shared research clusters.
- Potential engagement
- AI fabric observability assessment for an existing or planned PowerEdge GPU cluster: telemetry gap analysis, topology mapping, and a supervised pilot of agent-assisted diagnosis.
- Unsupported claims
- No measured GPU-utilization or job-time improvement is reported.
- No customer production deployment is cited.
- Closed-loop autonomous remediation is not demonstrated.
- Results are not claimed to generalize to other environments.
Pre-sales engineering
Role takeaway
- Fit
- Relevant to multi-host GPU clusters (e.g., PowerEdge XE9680 with NVIDIA or AMD GPUs) on RoCEv2 Ethernet fabrics such as PowerSwitch Z9864F-ON with Dell SONiC.
- Architecture and integration
- Requires exporters for NIC RDMA counters, switch telemetry, LLDP topology and iDRAC Redfish feeding a correlation layer (here Metrum AI Ops), with output routed to a ticketing system for human approval.
- Prerequisites
- RDMA NIC counter exporter
- Switch PFC/ECN/queue telemetry collection
- LLDP-based host-to-port topology map
- iDRAC Redfish access
- Ticketing/change-approval integration
- Constraints
- Single-incident lab evidence
- Counter sampling window and reset history not established
- ~60 s scrape floor bounds detection time
- Agent platform is a third-party product
- Remediation efficacy unmeasured
- Security
- Agents need scoped read-only credentials to iDRAC, switches and NICs; credential controls and change management are stated prerequisites for any autonomous action.
- Proof of value
- Replay or inject a RoCE receiver back-pressure scenario and measure detection and localization time
- Confirm alerts fire where existing dashboards stay silent
- Test receive-buffer and RNR retry tuning or collective-library admission on a representative training job
Delivery
Role takeaway
- Work
- Add fabric observability and agent-assisted diagnosis with a human approval boundary to AI cluster deployment and acceptance plans.
- Dependencies
- Telemetry exporters deployed on NICs and switches
- Accurate topology data
- Change-management process
- Partner tool licensing if Metrum AI Ops is used
- Ownership
- Research computing operations owns runbooks and approvals; network engineering owns fabric telemetry; vendor/partner supports tooling.
- Skills and adoption
- Operators need training on RoCE congestion mechanisms (ECN/DCQCN, PFC, RNR NAK) and on reviewing agent-generated root-cause recommendations.
- Governance checkpoints
- Read-only telemetry access approved
- Human approval required before any configuration change
- Rollback and audit procedures defined before closed-loop use
- Post-incident review of agent recommendations
- Acceptance criteria
- Cross-layer counters visible for all GPU hosts and leaf ports
- Injected fabric fault detected and localized within agreed time
- Recommendations logged to ticketing with evidence attached
- Risks
- Overtrusting LLM-generated diagnoses
- Credential sprawl from agent access
- Vendor lab results not reproducing on campus workloads
- Tactical tuning masking root causes
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Multi-host AI training fabrics need a NIC-to-switch-port topology map (LLDP) and collection of control-plane counters (RNR NAK, PFC pauses) alongside throughput and link state. The note recommends evaluating receiver-driven/clear-to-send admission in NCCL/RCCL co-tuned with the fabric, citing Meta's SIGCOMM 2024 experience, over switch-level DCQCN tuning alone.
Governance
Who approves, reviews and stays accountable for outcomes?
Agentic operations were bounded: telemetry read-only, output a recommendation, a human must approve any configuration change. The document states policy enforcement, automated rollback and audit integration were not exercised and are prerequisites for closed-loop operation.
Security and privacy
What data, permissions and controls need testing?
Agents reading iDRAC (Redfish), switch and NIC telemetry require scoped, read-only credentials; the disclaimer states autonomous remediation requires credential controls and change-management integration. No security testing of the agent platform is reported.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Small campus operations teams may benefit from assisted diagnosis, but must build RoCE and RDMA skills to validate agent output; no staffing impact is measured.
Procurement
What should contracts, pricing and exit terms secure?
When buying GPU clusters, specify telemetry exporters, topology mapping and operations tooling (and their licensing) as deliverables, not afterthoughts; treat partner agent platforms as separately evaluated software.
Operating model
Which teams own the service once it runs?
Shifts AI cluster operations toward cross-layer observability with agent-assisted triage under human approval; closed-loop remediation remains a future, governed phase.
Publication history
- 2026-11-07Dell · Issue 102 resources
Stable resource ID: dell-broadcom-pikes-peak-ai-lab-roce-gray-failure-agentic-rca-20261103