{"resourceId":"nvidia-blog-codesign-every-byte-ai-factory-dynamo-20261105","versions":[{"version":"external-224aa983b64cdbc5414d01036d0ec9819b6d25967f3c59739e70e42cf591706b","resource":{"id":"nvidia-blog-codesign-every-byte-ai-factory-dynamo-20261105","title":"NVIDIA co-design blog: Dynamo KV routing, disaggregation, and up to 30x throughput/MW claims for Vera Rubin NVL72","organization":"NVIDIA","sector":"AI infrastructure","geography":"Global (AI factory reference; vendor architecture claims)","publishedAt":"November 5, 2026","publicationDate":"2026-11-05","eventDate":null,"sourceName":"NVIDIA Technical Blog","sourceLabel":"NVIDIA Technical Blog","sourceUrl":"https://developer.nvidia.com/blog/five-ways-nvidia-co-design-maximizes-the-value-of-every-byte-in-the-ai-factory/","evidenceClass":"vendor-claim","outcomeClass":"emerging","topics":["infrastructure","developers-agents","operating-model"],"finding":"NVIDIA's Nov 5 co-design post positions five levers—NVFP4, wide expert parallelism, Dynamo KV-aware routing/disaggregated serving/KV Cache Runner via NIXL, Vera CPU staging, and CMX/BlueField-4 storage—as a stacked memory system, claiming Vera Rubin NVL72 can deliver up to 30x throughput per megawatt versus GB300 NVL72 from the same 288 GB HBM per GPU.","sledRelevance":"SLED AI-factory planners see Dynamo and memory-tier messaging in NVIDIA roadmaps; this post is the vendor architecture narrative those RFPs will cite. Pair it with independent Dynamo measurements (filed in this edition) rather than treating 30x/MW or fleet-wide KV reuse as transferable campus outcomes.","evidence":"Body states Dynamo treats KV cache as a fleet-wide resource rather than a per-GPU side effect via KV-aware routing, disaggregated serving, and KV Cache Runner. Cites VAST/WEKA NIXL validation throughput figures. Summary and body repeat up to 30x throughput per megawatt versus GB300 NVL72 at equal 288 GB/GPU.","architectureImplications":"Assumes fast KV transfer (NIXL/RDMA/GPUDirect-class paths), prefix reuse worth caching, and Rubin/Vera/CMX stack availability. Campus sites on Ethernet-only or mixed GPU generations will not inherit the stacked gains without fabric and software parity.","governanceImplications":"Require vendors to map each lever to what the site will actually deploy in year one; disallow 30x/MW marketing figures as acceptance criteria without site-defined tokens/watt tests.","securityPrivacyImplications":"Fleet-wide KV cache and external storage tiers expand where prompt/context state lives; residency, tenant isolation, and eviction policy become governance items for shared university or state pools.","caveats":"Pure vendor architecture claim. Throughput/MW figure is NVIDIA comparison under its agentic assumptions. Storage vendor validation numbers are not campus production measurements. Independent Dynamo tests in this edition show routing and disaggregation benefits are workload- and link-dependent.","streamIds":["nvidia"],"roles":{"sales":"Frame this as NVIDIA's memory co-design story for AI factories, not as a measured SLED outcome. The post claims Dynamo makes KV cache fleet-wide and cites up to 30x throughput per megawatt for Vera Rubin NVL72 versus GB300. Use it to ask which levers the customer can actually field this year—routing software versus Rubin/Vera hardware—and to set expectations that independent tests show KV routing and disaggregation gains depend on prefix reuse and fabric speed.","engineering":"Treat Dynamo KV-aware routing, disaggregated prefill/decode, and KV Cache Runner as optional inference-plane features that need fabric proof. Validate NIXL transfer rates on the site's network before enabling disaggregation. Design for prefix-reuse workloads only; keep round-robin as fallback. Do not architect campus acceptance around the 30x/MW headline.","delivery":"If a pilot is warranted, stage Dynamo routing in monitor-and-measure mode on one inference pool with known shared prefixes, capture TTFT, error/503 rates, and KV hit proxies, then decide on disaggregation only after link throughput math clears. Document ownership between platform SRE and model owners for cache residency."},"roleDetails":{"sales":{"customerProblem":"AI-factory buyers hear stacked NVIDIA memory claims but lack a way to separate software levers from next-gen hardware.","stakeholders":["CIO/CTO","Research computing","Data-center facilities","Procurement"],"discoveryQuestions":["Which Dynamo features could you run on today's GPUs and network?","Do agent workloads share long prefixes worth caching?","What tokens-per-watt or PUE evidence will procurement accept?"],"valueHypothesis":"Selective Dynamo routing on reuse-heavy workloads may cut prefill waste if fabric and measurement gates are met—hardware 30x claims remain unverified for SLED sites.","potentialEngagement":"Inference-plane readiness assessment: routing pilot design plus fabric KV-transfer estimate.","unsupportedClaims":["30x throughput per megawatt on customer sites","Automatic gains without prefix reuse","Campus production references in this post"]},"engineering":{"fit":"Partial fit for Kubernetes inference pools with measurable prefix reuse and fast east-west fabric.","architectureIntegration":"Dynamo router modes; optional prefill/decode split; NIXL to CPU/SSD/network KV tiers.","prerequisites":["Measured inter-node bandwidth for KV bytes","Workload trace with prefix overlap","Observability for TTFT and 503/reject rates"],"constraints":["Vendor 30x/MW assumes Vera Rubin NVL72 stack","Disaggregation harmful on slow links","External KV storage only helps if reload beats prefill"],"security":"Define which tenants' KV may land on shared CPU/SSD tiers and retention/eviction rules.","proofOfValue":["A/B TTFT with kv vs round-robin on site trace","KV transfer time vs recomputed prefill for typical prompt sizes"]},"delivery":{"work":"Pilot KV-aware routing only; defer disaggregation until fabric math passes.","dependencies":["Inference platform ownership","Access to representative traces","Change window without production SLA risk"],"ownership":"Platform SRE owns router config; app teams own prompt templates that create reuse.","skillsAdoption":"Train on Dynamo router modes, NIXL basics, and saturation failure modes.","governanceCheckpoints":["Approve enabling disaggregated serving","Approve external KV storage residency"],"acceptanceCriteria":["Proposal: ≥20% TTFT improvement on reuse workload without material error-rate increase","Proposal: documented rollback to round-robin"],"risks":["Marketing 30x figures become informal KPIs","Silent routing degradation without hit-rate monitors","Disaggregation on Ethernet tanks TTFT"]}},"retrievedAt":"2026-11-07T04:01:22Z","enrichedAt":"2026-11-07T04:01:22Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Operating Dynamo, NIXL tiers, and MoE wide-EP needs specialized inference platform skills often scarce in municipal IT and mid-size campus shops.","procurementImplications":"Split RFP scoring between (a) HBM/GPU count and (b) verified KV-transfer bandwidth, routing mode behavior under load, and energy metrics for the specific facility.","operatingModelImplications":"If adopting Dynamo features, define prefix-reuse SLOs, saturation/503 handling policy, and when disaggregated serving is allowed given measured link throughput.","sourceVerification":{"openedUrl":"https://developer.nvidia.com/blog/five-ways-nvidia-co-design-maximizes-the-value-of-every-byte-in-the-ai-factory/","referenceExcerpt":"Dynamo is the distributed serving layer that closes this gap. It treats KV cache as a fleet-wide resource rather than a per-GPU side effect","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}