From the NVIDIA edition of November 6, 2026
NVIDIA co-design blog: Dynamo KV routing, disaggregation, and up to 30x throughput/MW claims for Vera Rubin NVL72
NVIDIA · AI infrastructure · Global (AI factory reference; vendor architecture claims)
- Publisher
- NVIDIA Technical Blog
- Original publication
- November 5, 2026
- Source retrieved
- 2026-11-07
What happened
NVIDIA's Nov 5 co-design post positions five levers—NVFP4, wide expert parallelism, Dynamo KV-aware routing/disaggregated serving/KV Cache Runner via NIXL, Vera CPU staging, and CMX/BlueField-4 storage—as a stacked memory system, claiming Vera Rubin NVL72 can deliver up to 30x throughput per megawatt versus GB300 NVL72 from the same 288 GB HBM per GPU.
Why it matters
SLED AI-factory planners see Dynamo and memory-tier messaging in NVIDIA roadmaps; this post is the vendor architecture narrative those RFPs will cite. Pair it with independent Dynamo measurements (filed in this edition) rather than treating 30x/MW or fleet-wide KV reuse as transferable campus outcomes.
Evidence and measured results
Body states Dynamo treats KV cache as a fleet-wide resource rather than a per-GPU side effect via KV-aware routing, disaggregated serving, and KV Cache Runner. Cites VAST/WEKA NIXL validation throughput figures. Summary and body repeat up to 30x throughput per megawatt versus GB300 NVL72 at equal 288 GB/GPU.
Limitations and uncertainty
Pure vendor architecture claim. Throughput/MW figure is NVIDIA comparison under its agentic assumptions. Storage vendor validation numbers are not campus production measurements. Independent Dynamo tests in this edition show routing and disaggregation benefits are workload- and link-dependent.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-07; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- AI-factory buyers hear stacked NVIDIA memory claims but lack a way to separate software levers from next-gen hardware.
- Stakeholders
- CIO/CTO
- Research computing
- Data-center facilities
- Procurement
- Discovery questions
- Which Dynamo features could you run on today's GPUs and network?
- Do agent workloads share long prefixes worth caching?
- What tokens-per-watt or PUE evidence will procurement accept?
- Value hypothesis
- Selective Dynamo routing on reuse-heavy workloads may cut prefill waste if fabric and measurement gates are met—hardware 30x claims remain unverified for SLED sites.
- Potential engagement
- Inference-plane readiness assessment: routing pilot design plus fabric KV-transfer estimate.
- Unsupported claims
- 30x throughput per megawatt on customer sites
- Automatic gains without prefix reuse
- Campus production references in this post
Pre-sales engineering
Role takeaway
- Fit
- Partial fit for Kubernetes inference pools with measurable prefix reuse and fast east-west fabric.
- Architecture and integration
- Dynamo router modes; optional prefill/decode split; NIXL to CPU/SSD/network KV tiers.
- Prerequisites
- Measured inter-node bandwidth for KV bytes
- Workload trace with prefix overlap
- Observability for TTFT and 503/reject rates
- Constraints
- Vendor 30x/MW assumes Vera Rubin NVL72 stack
- Disaggregation harmful on slow links
- External KV storage only helps if reload beats prefill
- Security
- Define which tenants' KV may land on shared CPU/SSD tiers and retention/eviction rules.
- Proof of value
- A/B TTFT with kv vs round-robin on site trace
- KV transfer time vs recomputed prefill for typical prompt sizes
Delivery
Role takeaway
- Work
- Pilot KV-aware routing only; defer disaggregation until fabric math passes.
- Dependencies
- Inference platform ownership
- Access to representative traces
- Change window without production SLA risk
- Ownership
- Platform SRE owns router config; app teams own prompt templates that create reuse.
- Skills and adoption
- Train on Dynamo router modes, NIXL basics, and saturation failure modes.
- Governance checkpoints
- Approve enabling disaggregated serving
- Approve external KV storage residency
- Acceptance criteria
- Proposal: ≥20% TTFT improvement on reuse workload without material error-rate increase
- Proposal: documented rollback to round-robin
- Risks
- Marketing 30x figures become informal KPIs
- Silent routing degradation without hit-rate monitors
- Disaggregation on Ethernet tanks TTFT
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Assumes fast KV transfer (NIXL/RDMA/GPUDirect-class paths), prefix reuse worth caching, and Rubin/Vera/CMX stack availability. Campus sites on Ethernet-only or mixed GPU generations will not inherit the stacked gains without fabric and software parity.
Governance
Who approves, reviews and stays accountable for outcomes?
Require vendors to map each lever to what the site will actually deploy in year one; disallow 30x/MW marketing figures as acceptance criteria without site-defined tokens/watt tests.
Security and privacy
What data, permissions and controls need testing?
Fleet-wide KV cache and external storage tiers expand where prompt/context state lives; residency, tenant isolation, and eviction policy become governance items for shared university or state pools.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Operating Dynamo, NIXL tiers, and MoE wide-EP needs specialized inference platform skills often scarce in municipal IT and mid-size campus shops.
Procurement
What should contracts, pricing and exit terms secure?
Split RFP scoring between (a) HBM/GPU count and (b) verified KV-transfer bandwidth, routing mode behavior under load, and energy metrics for the specific facility.
Operating model
Which teams own the service once it runs?
If adopting Dynamo features, define prefix-reuse SLOs, saturation/503 handling policy, and when disaggregated serving is allowed given measured link throughput.
Publication history
- 2026-11-06NVIDIA · Issue 626 resources
Stable resource ID: nvidia-blog-codesign-every-byte-ai-factory-dynamo-20261105