{"resourceId":"dell-broadcom-pikes-peak-ai-lab-roce-gray-failure-agentic-rca-20261103","versions":[{"version":"external-3b33ccd81d0c72bed9520dde8051bd4bc43a4275edf817eb1afc2845cf34b821","resource":{"id":"dell-broadcom-pikes-peak-ai-lab-roce-gray-failure-agentic-rca-20261103","title":"Dell InfoHub field note: in the Dell/Broadcom Pikes Peak AI Lab, Metrum AI Ops agents correlated NIC, PowerSwitch and topology telemetry to localize a silent RoCE receiver back-pressure \"gray failure\" in about five minutes, with remediation staged for human approval","organization":"Dell Technologies, Broadcom and Metrum AI (Pikes Peak AI Lab, a Dell Technologies and Broadcom facility)","sector":"AI infrastructure operations (vendor validation lab); applicable to university and research-computing GPU clusters","geography":"Pikes Peak AI Lab (Dell/Broadcom facility; location not stated in the document); U.S. vendor lab, not a customer deployment","publishedAt":"2026-11-03","publicationDate":"2026-11-03","eventDate":null,"sourceName":"Dell Technologies InfoHub","sourceLabel":"Vendor technical field note / case study (Dell, Broadcom, Metrum AI)","sourceUrl":"https://www.dell.com/en-us/infohub/articles/from-gray-failure-to-root-cause-autonomous-operations-in-the-pikes-peak-ai-lab","evidenceClass":"vendor-claim","outcomeClass":"emerging","topics":["infrastructure","operating-model","developers-agents"],"finding":"A Nov. 3, 2026 Dell InfoHub field note describes a 23-host, 166-GPU lab (PowerEdge XE9680 with AMD MI300X and NVIDIA H200, PowerSwitch Z9864F-ON on Dell SONiC 4.4.0, Broadcom Thor 2 RoCE NICs, PowerScale F910) where switch queue drops were zero and dashboards raised no actionable alert, yet NIC counters showed ~7.3M RNR NAK retries and ~4.48% of packets resent. Metrum AI Ops agents correlated NIC, switch and LLDP topology data, ruled out look-alikes and produced a ranked recommendation in ~5 minutes; remediation was staged for human approval and not applied.","sledRelevance":"University research-computing centers and state-funded AI platforms increasingly run multi-host GPU training over RoCE Ethernet fabrics with small operations teams. This note shows a failure mode standard dashboards miss and a read-only, human-approved agent workflow for diagnosing it. Relevance is engineering and operations; it is a vendor lab, not a public-sector deployment, and no SLED customer is named.","evidence":"Vendor-authored single-incident lab investigation. Reported measurements: ~7,308,289 RNR NAK retries and ~7,305,209 receive-side discards on one host; ~4.48% retransmission share; MTTD ~45–90 s, MTTK +~30–90 s, ~5 min end-to-end (marked \"Measured\"). The document states counter sampling window, reset history and PFC direction were not established, and that it is not a benchmark or guarantee.","architectureImplications":"Multi-host AI training fabrics need a NIC-to-switch-port topology map (LLDP) and collection of control-plane counters (RNR NAK, PFC pauses) alongside throughput and link state. The note recommends evaluating receiver-driven/clear-to-send admission in NCCL/RCCL co-tuned with the fabric, citing Meta's SIGCOMM 2024 experience, over switch-level DCQCN tuning alone.","governanceImplications":"Agentic operations were bounded: telemetry read-only, output a recommendation, a human must approve any configuration change. The document states policy enforcement, automated rollback and audit integration were not exercised and are prerequisites for closed-loop operation.","securityPrivacyImplications":"Agents reading iDRAC (Redfish), switch and NIC telemetry require scoped, read-only credentials; the disclaimer states autonomous remediation requires credential controls and change-management integration. No security testing of the agent platform is reported.","caveats":"Single lab incident in a Dell/Broadcom facility using a partner platform (Metrum AI Ops); vendor-authored and promotional framing (\"practical validation of the Dell AI Factory operating model\"). No independent replication, no workload slowdown magnitude, no duration of the fault, and remediation not applied. The InfoHub page is dated Nov. 3, 2026 (PDF metadata shows creation Sept. 25, 2026). Published before the last completed run; included because it was not archived and adds operational-constraint evidence not covered in prior Dell editions.","streamIds":["dell","research"],"roles":{"sales":"Sales should position operations readiness, not hardware, as the conversation. This Dell field note shows a GPU fabric that looked healthy while roughly 1 in 22 packets were being resent, with no actionable alert from standard dashboards. For research computing leaders, the question is whether their team could see and explain such a slowdown. Use it to open discovery on observability and staffing, while being clear it is a vendor lab result with a partner tool, not a customer-proven outcome or a Dell product guarantee.","engineering":"Engineering should read this as a telemetry-architecture lesson. Diagnosing the RoCE back-pressure required correlating Broadcom NIC RDMA counters, PowerSwitch PFC and queue counters, and an LLDP host-to-port map, then using iDRAC to rule out host faults. Teams running XE9680-class clusters should verify they export RNR NAK and PFC counters, maintain topology mapping, and evaluate collective-library admission control. The ~5 minute diagnosis covers detection and root cause only; slowdown magnitude and fix efficacy were not measured.","delivery":"Delivery should treat AI cluster acceptance as including operability, not just installation. The note frames gray failures as invisible to standard monitoring and the agent workflow as read-only with human approval, explicitly not demonstrating rollback, policy enforcement or audit integration. Delivery plans for research clusters should include telemetry exporters, topology mapping, runbooks for RoCE congestion classes, and a change-approval boundary before any agent-assisted remediation, with acceptance tests that inject or replay fabric faults."},"roleDetails":{"sales":{"customerProblem":"Research computing teams running multi-host GPU training over RoCE fabrics can lose GPU time to faults that leave links up, drops at zero and dashboards quiet, as this Dell lab incident shows.","stakeholders":["University research computing director","HPC/AI cluster network engineers","CIO or CTO funding AI infrastructure","Principal investigators reliant on shared GPU clusters"],"discoveryQuestions":["Do you currently collect RNR NAK and PFC pause counters from NICs and switches?","Can you map each host NIC to its switch port automatically?","How do you detect training jobs slowed by fabric back-pressure today?","What approval process governs configuration changes on the cluster fabric?","Would you allow read-only agent access to iDRAC and switch telemetry?"],"valueHypothesis":"Cross-layer telemetry correlation may convert undetected fabric slowdowns into diagnosed incidents within minutes, protecting GPU utilization on shared research clusters.","potentialEngagement":"AI fabric observability assessment for an existing or planned PowerEdge GPU cluster: telemetry gap analysis, topology mapping, and a supervised pilot of agent-assisted diagnosis.","unsupportedClaims":["No measured GPU-utilization or job-time improvement is reported.","No customer production deployment is cited.","Closed-loop autonomous remediation is not demonstrated.","Results are not claimed to generalize to other environments."]},"engineering":{"fit":"Relevant to multi-host GPU clusters (e.g., PowerEdge XE9680 with NVIDIA or AMD GPUs) on RoCEv2 Ethernet fabrics such as PowerSwitch Z9864F-ON with Dell SONiC.","architectureIntegration":"Requires exporters for NIC RDMA counters, switch telemetry, LLDP topology and iDRAC Redfish feeding a correlation layer (here Metrum AI Ops), with output routed to a ticketing system for human approval.","prerequisites":["RDMA NIC counter exporter","Switch PFC/ECN/queue telemetry collection","LLDP-based host-to-port topology map","iDRAC Redfish access","Ticketing/change-approval integration"],"constraints":["Single-incident lab evidence","Counter sampling window and reset history not established","~60 s scrape floor bounds detection time","Agent platform is a third-party product","Remediation efficacy unmeasured"],"security":"Agents need scoped read-only credentials to iDRAC, switches and NICs; credential controls and change management are stated prerequisites for any autonomous action.","proofOfValue":["Replay or inject a RoCE receiver back-pressure scenario and measure detection and localization time","Confirm alerts fire where existing dashboards stay silent","Test receive-buffer and RNR retry tuning or collective-library admission on a representative training job"]},"delivery":{"work":"Add fabric observability and agent-assisted diagnosis with a human approval boundary to AI cluster deployment and acceptance plans.","dependencies":["Telemetry exporters deployed on NICs and switches","Accurate topology data","Change-management process","Partner tool licensing if Metrum AI Ops is used"],"ownership":"Research computing operations owns runbooks and approvals; network engineering owns fabric telemetry; vendor/partner supports tooling.","skillsAdoption":"Operators need training on RoCE congestion mechanisms (ECN/DCQCN, PFC, RNR NAK) and on reviewing agent-generated root-cause recommendations.","governanceCheckpoints":["Read-only telemetry access approved","Human approval required before any configuration change","Rollback and audit procedures defined before closed-loop use","Post-incident review of agent recommendations"],"acceptanceCriteria":["Cross-layer counters visible for all GPU hosts and leaf ports","Injected fabric fault detected and localized within agreed time","Recommendations logged to ticketing with evidence attached"],"risks":["Overtrusting LLM-generated diagnoses","Credential sprawl from agent access","Vendor lab results not reproducing on campus workloads","Tactical tuning masking root causes"]}},"retrievedAt":"2026-11-08T04:04:12.065Z","enrichedAt":"2026-11-08T04:08:38.894Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Small campus operations teams may benefit from assisted diagnosis, but must build RoCE and RDMA skills to validate agent output; no staffing impact is measured.","procurementImplications":"When buying GPU clusters, specify telemetry exporters, topology mapping and operations tooling (and their licensing) as deliverables, not afterthoughts; treat partner agent platforms as separately evaluated software.","operatingModelImplications":"Shifts AI cluster operations toward cross-layer observability with agent-assisted triage under human approval; closed-loop remediation remains a future, governed phase.","sourceVerification":{"openedUrl":"https://www.dell.com/en-us/infohub/articles/from-gray-failure-to-root-cause-autonomous-operations-in-the-pikes-peak-ai-lab","referenceExcerpt":"correlated compute, network, and storage telemetry to identify a silent RoCE back-pressure condition and stage a governed remediation","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}