Strategic PartnersIssue 626 records
NVIDIA
NVIDIA published a Nov 6 NeMo RL Reshard collective post with 2.7x–5.7x weight-refit speedups on DGX H100/GB200, while its Nov 5 AI-factory co-design blog elevates Dynamo KV routing and disaggregation inside a stacked memory story claiming up to 30x throughput per megawatt for Vera Rubin NVL72. Independent measurements caution that Dynamo KV routing near saturation can shed requests with HTTP 503s, and that disaggregated serving on 1GbE-class links can be more than 20× slower as TTFT follows KV bytes over link rate. On the SLED operating side, Utah CHPC’s November newsletter moves the statewide Redtail AI Factory into a published Winter 2027 allocation calendar (requests Nov 9–Dec 1; 15,000 GPU-hours/quarter), and Accruex restates facility-level PUE/WUE diligence buyers should demand beside vendor efficiency narratives.
- Evidence records
- 6
- Cross-source patterns
- 2
- Evidence classes
- 3 independent research2 vendor claim1 public-sector association guidance
- Outcomes
- 3 emerging3 cautionary
- Source freshness
- 2 new this fortnight2 recent1 older, newly relevant1 undated
- Research completed
- 2026-11-07
Choose a role to see its takeaway beside every record in the ledger.
Patterns across the evidence
Synthesis · Lighthouse Advisory interpretation
NVIDIA Dynamo KV/disaggregation claims need fabric proof and an overload policy—independent tests show 503 shedding and link-bound slowdowns
NVIDIA's Nov 5 co-design post presents Dynamo KV-aware routing, disaggregated serving, and tiered KV reuse as core AI-factory memory levers. Michele Campi's June 2026 saturation study found the KV-router near overload returned 503s for roughly one in seven requests to protect latency, while round-robin completed all calls. TT Lab's October 2026 hands-on saw KV routing help repeated prefixes but measured more than 20× slower TTFT for disaggregation across two servers on a 1GbE-class link, matching KV bytes over link rate. Together they tell SLED teams to pilot routing with hit-rate and reject-rate metrics, and to withhold disaggregation until transfer math clears.
Operating questionBefore enabling Dynamo KV routing or disaggregated serving on a shared SLED inference pool, what measured east-west KV throughput and maximum acceptable 503/reject rate will gate go-live?
Supporting evidenceNVIDIA co-design blog: Dynamo KV routing, disaggregation, and up to 30x throughput/MW claims for Vera Rubin NVL72Michele Campi (independent)TT Lab (Toss ML/AI Infra series)
Shared AI factories need published allocation rules and facility-level energy diligence—not only vendor throughput-per-watt narratives
Utah CHPC's November newsletter moves Redtail from launch theater to operating practice with a Nov 9–Dec 1, 2026 request window, Jan 1, 2027 award start, and 15,000 GPU-hour quarterly caps on a project basis. Accruex's buyer guide insists efficiency claims be tied to the specific facility's PUE/WUE, clean-energy coverage, and comparable power benchmarks rather than fleet averages or chip slides. Read beside NVIDIA's co-design throughput-per-megawatt messaging, the pair argues SLED buyers should score access governance and facility evidence as first-class acceptance items.
Operating questionFor the next state or campus AI-factory go-live, which allocation calendar/caps and facility PUE/WUE disclosures will be published before efficiency marketing is used externally?
Supporting evidenceUniversity of Utah Center for High Performance Computing (CHPC)AccruexNVIDIA co-design blog: Dynamo KV routing, disaggregation, and up to 30x throughput/MW claims for Vera Rubin NVL72
Evidence ledger
Full record · every source keeps its link and limitations
NVIDIA Reshard collective: NeMo RL weight refit 2.7x–5.7x faster than gather-and-broadcast on DGX H100/GB200
NVIDIA's Nov 6 post describes NCCL M2N Reshard as a DTensor-to-DTensor weight-refit path for disaggregated RL post-training, reporting 2.7x–5.7x lower refit latency versus gather-and-broadcast in NeMo RL on DGX H100 and GB200 across models including Qwen 3 235B and DeepSeek v3 671B, with hybrid Reshard for dominant FFN weights.
Why it matters, evidence and limitations
- Why it matters
- Campus and state AI factories that run RL or continual post-training on shared GPU pools face refit as a recurring communication tax when training and inference are disaggregated. The post is vendor-measured but gives concrete layouts and latencies SLED platform teams can use as a starting hypothesis—not a procurement guarantee—when sizing interconnect and scheduling between training and inference partitions.
- Evidence and measured results
- Quoted NeMo RL benchmarks: e.g. DeepSeek v3 671B on DGX H100 gather-and-broadcast 14.9 s vs Reshard 2.6 s; Qwen 3 235B 6.4 s vs 1.9 s. States payload can reach ~10 TB per refit for largest open-weight models. Configuration snippet sets refit_transport: nccl_reshard with colocated generation disabled.
- Limitations and uncertainty
- Vendor blog; results are NeMo RL reference measurements on NVIDIA DGX systems, not third-party replication. Hybrid path still uses gather-and-broadcast for some weights. NeMo RL refit guide URL cited in post returned 404 when checked during this run.
NVIDIA co-design blog: Dynamo KV routing, disaggregation, and up to 30x throughput/MW claims for Vera Rubin NVL72
NVIDIA's Nov 5 co-design post positions five levers—NVFP4, wide expert parallelism, Dynamo KV-aware routing/disaggregated serving/KV Cache Runner via NIXL, Vera CPU staging, and CMX/BlueField-4 storage—as a stacked memory system, claiming Vera Rubin NVL72 can deliver up to 30x throughput per megawatt versus GB300 NVL72 from the same 288 GB HBM per GPU.
Why it matters, evidence and limitations
- Why it matters
- SLED AI-factory planners see Dynamo and memory-tier messaging in NVIDIA roadmaps; this post is the vendor architecture narrative those RFPs will cite. Pair it with independent Dynamo measurements (filed in this edition) rather than treating 30x/MW or fleet-wide KV reuse as transferable campus outcomes.
- Evidence and measured results
- Body states Dynamo treats KV cache as a fleet-wide resource rather than a per-GPU side effect via KV-aware routing, disaggregated serving, and KV Cache Runner. Cites VAST/WEKA NIXL validation throughput figures. Summary and body repeat up to 30x throughput per megawatt versus GB300 NVL72 at equal 288 GB/GPU.
- Limitations and uncertainty
- Pure vendor architecture claim. Throughput/MW figure is NVIDIA comparison under its agentic assumptions. Storage vendor validation numbers are not campus production measurements. Independent Dynamo tests in this edition show routing and disaggregation benefits are workload- and link-dependent.
Independent test: Dynamo KV-router under load returns 503s (~1 in 7) to keep latency low—trade-off, not pure win
Independent A/B on Dynamo with Mooncake-trace load shows the KV-aware router is not simply faster: near saturation (N=2 workers on the author's 8×A100 box) it shed ~14.3% of requests with HTTP 503 ResourceExhausted while round-robin completed 100%, trading rejects for better latency among accepted calls; the gap vanished with more workers.
Why it matters, evidence and limitations
- Why it matters
- Direct counterweight to NVIDIA Dynamo KV-routing marketing for shared campus or municipal inference gateways that cannot silently drop citizen or researcher requests. SLED operators must decide whether latency for accepted traffic or completion rate is the governing SLO before enabling kv router mode.
- Evidence and measured results
- Author reports at N=2: kv-router completed 2571/3000 (429 failures, 14.3%) vs round-robin 3000/3000; failures are explicit 503 ResourceExhausted. At N=4 and N=8 both arms completed 100%. Workload is Mooncake production-style trace with prefix sharing; Dynamo v1.2.0 pinned.
- Limitations and uncertainty
- Single-author lab on one 8×A100 box; Dynamo v1.2.0 may differ from later releases. Benefit profile is saturation-specific. Not a SLED production deployment.
TT Lab hands-on: Dynamo KV routing works; disaggregated serving >20× slower across two servers on 1GbE-class link
Draft hands-on (Dynamo 1.6.0.dev, Qwen3-0.6B) finds KV-aware routing reliably steers repeated prefixes (up to 60/60 hits vs ~35–43 round-robin) but warns of silent degradation when KV events are missing; disaggregated serving was 4.7–7.2× slower on one shared GPU and more than 20× slower across two servers where TTFT matched KV bytes divided by ~104 MB/s link throughput. NVLink/RDMA benefits remain unmeasured.
Why it matters, evidence and limitations
- Why it matters
- Many SLED inference pilots still sit on enterprise Ethernet, not RDMA AI fabrics. This measurement shows disaggregated prefill/decode can be link-bound and harmful in that setting, while KV routing can still help—if event publishing is verified. Use as a pre-flight checklist beside NVIDIA Dynamo claims.
- Evidence and measured results
- Author tables: cross-server 1,500-word TTFT ~1.7 s vs standalone tens of ms; 3,000-word ~3.4–3.5 s; transfer ~351.75 MB at ~104 MB/s. KV event-disabled mode fell to random hit rates with only a log warning. Status marked Draft 2026-10-07; limitations section lists unmeasured RDMA/NVLink.
- Limitations and uncertainty
- Draft post; tiny 0.6B model; synthetic prefixes; concurrency limited; dev build may differ from GA. Worst-case 1GbE-class link by design. Not peer-reviewed.
Utah CHPC: Redtail AI Factory formally launched; Winter 2027 allocations open Nov 9 (15k GPU-hours/quarter)
CHPC's November 2026 newsletter states Redtail AI Factory formally launched Oct 6, 2026, will move to standard quarterly allocations beginning Jan 1, 2027, opens request window Nov 9–Dec 1, 2026 for Winter 2027, and caps new projects at 15,000 GPU hours and/or 100,000 CPU core hours per quarter, allocated by project rather than research group.
Why it matters, evidence and limitations
- Why it matters
- Concrete U.S. state/university AI-factory operating model: public–private build (prior archive noted NVIDIA/HPE) now publishing allocation policy, timelines, and project-based governance. Transferable pattern for other states standing up shared GPU factories—limits: Utah-specific portal process; no performance outcomes yet.
- Evidence and measured results
- Launch date Oct 6 at One-U Responsible AI symposium; allocation start Jan 1, 2027; request window Nov 9–Dec 1, 2026; quarterly limit 15,000 GPU hours and/or 100,000 CPU core hours; CPU-only jobs allowed using leftover cores (40 of 104 per node after 64 assigned to GPUs); challenge projects inherit prior six-month allocations.
- Limitations and uncertainty
- Operational announcement, not an outcome evaluation. Hardware counts and NVIDIA role are not restated in this newsletter (covered in earlier archived Redtail materials). 'Last Updated' footer noise on page; substance is the allocation calendar.
Accruex: GPU buyers should demand facility-level PUE/WUE, clean-energy evidence, and comparable power benchmarks
Accruex's Sept 15, 2026 buyer guide argues sustainability claims for AI compute must be facility-specific: ask for PUE and WUE of the site that will run the workload, standards and coverage behind clean-energy claims, comparability to MLPerf Power or AI Energy Score, emissions accounting including material Scope 3, and whether total energy for the workload class is rising or falling—not fleet-wide averages or chip-only efficiency stories.
Why it matters, evidence and limitations
- Why it matters
- Pairs with NVIDIA AI-factory efficiency marketing (including throughput-per-megawatt claims). U.S. SLED procurement can put these five questions into RFPs for on-prem or colocation GPU capacity. Limits: guidance essay, not an audit of NVIDIA or a specific Utah/municipal factory.
- Evidence and measured results
- Lists five due-diligence questions covering facility PUE/WUE, clean-energy standard/share, independent efficiency benchmarks, emissions boundary/Scope 3, and year-over-year total energy for the workload class. Emphasizes separating GPU efficiency from facility efficiency.
- Limitations and uncertainty
- Commercial advisory content; not a standards body. Does not validate any named provider's numbers. MLPerf Power / AI Energy Score coverage may not match a buyer's exact model.
How to read this edition
Source findings, measured results and limitations come from the cited publications. Patterns, operating questions, role takeaways and implementation considerations are Lighthouse Advisory interpretation, stated as questions to validate locally rather than guaranteed outcomes. Vendor and operator claims are labeled as claims. Full research method.
- Independent research
- Research conducted outside the implementing organization.
- Vendor claim
- A supplier-provided assertion that has not been upgraded to independent evidence.
- Public-sector association guidance
- Practitioner guidance or an association-supplied case; not independent outcome evidence.