From the NVIDIA edition of November 6, 2026
TT Lab hands-on: Dynamo KV routing works; disaggregated serving >20× slower across two servers on 1GbE-class link
TT Lab (Toss ML/AI Infra series) · Independent AI infrastructure research · Author lab (single 8GB-class GPU + second consumer GPU; 1GbE-class inter-server)
- Publisher
- TT Lab Blog
- Original publication
- October 7, 2026
- Source retrieved
- 2026-11-07
What happened
Draft hands-on (Dynamo 1.6.0.dev, Qwen3-0.6B) finds KV-aware routing reliably steers repeated prefixes (up to 60/60 hits vs ~35–43 round-robin) but warns of silent degradation when KV events are missing; disaggregated serving was 4.7–7.2× slower on one shared GPU and more than 20× slower across two servers where TTFT matched KV bytes divided by ~104 MB/s link throughput. NVLink/RDMA benefits remain unmeasured.
Why it matters
Many SLED inference pilots still sit on enterprise Ethernet, not RDMA AI fabrics. This measurement shows disaggregated prefill/decode can be link-bound and harmful in that setting, while KV routing can still help—if event publishing is verified. Use as a pre-flight checklist beside NVIDIA Dynamo claims.
Evidence and measured results
Author tables: cross-server 1,500-word TTFT ~1.7 s vs standalone tens of ms; 3,000-word ~3.4–3.5 s; transfer ~351.75 MB at ~104 MB/s. KV event-disabled mode fell to random hit rates with only a log warning. Status marked Draft 2026-10-07; limitations section lists unmeasured RDMA/NVLink.
Limitations and uncertainty
Draft post; tiny 0.6B model; synthetic prefixes; concurrency limited; dev build may differ from GA. Worst-case 1GbE-class link by design. Not peer-reviewed.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-07; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Teams copy Dynamo disaggregation patterns without checking whether their network can move KV cache in time.
- Stakeholders
- Network architecture
- Inference platform
- CIO for AI factory pilots
- Discovery questions
- What is measured bandwidth between potential prefill and decode nodes?
- Are you proposing disaggregation or only KV-aware routing?
- How will you detect silent KV-event failures?
- Value hypothesis
- Routing-only pilots on reuse-heavy traffic may pay off; disaggregation should wait for fabric proof.
- Potential engagement
- Fabric-and-routing readiness checklist workshop before Dynamo feature enablement.
- Unsupported claims
- That 0.6B results scale linearly to 70B+
- That RDMA would have shown gains here (unmeasured)
- Peer-reviewed status
Pre-sales engineering
Role takeaway
- Fit
- Strong cautionary fit for Ethernet campus inference; partial positive for KV routing.
- Architecture and integration
- Dynamo router modes; NIXL KV transfer paths; avoid disagg until link math clears.
- Prerequisites
- Ability to capture frontend KV-event warnings
- Measured inter-node throughput
- Same-prefix TTFT probe
- Constraints
- Draft + dev build
- Tiny model / synthetic prefixes
- RDMA/NVLink unmeasured
- Security
- Encrypt and isolate cross-node KV transfer; review multi-server blast radius.
- Proof of value
- Hit-rate improvement with events enabled
- Disagg TTFT estimate within agreed budget before enable
Delivery
Role takeaway
- Work
- Routing pilot with event-health checks; disagg only if transfer estimate passes.
- Dependencies
- Network team measurement support
- Non-production Dynamo namespace
- Stakeholder agreement on draft-source limits
- Ownership
- Platform engineering owns Dynamo flags; networking owns fabric evidence.
- Skills and adoption
- Teach KV bytes/token estimation and log warning triage.
- Governance checkpoints
- Approve disaggregated serving enablement
- Review silent-failure monitoring
- Acceptance criteria
- Proposal: no disagg enable without measured ≥X GB/s path
- Proposal: alert on kv_event_publisher_disabled warnings
- Risks
- Stakeholders dismiss draft as non-authoritative
- Teams still enable disagg from vendor diagrams
- Dev/GA flag drift
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Estimate added TTFT ≈ (tokens × KV bytes/token) / measured transfer throughput before enabling disaggregation. Prefer KV routing pilots with event-health checks over disaggregation on slow links.
Governance
Who approves, reviews and stays accountable for outcomes?
Require vendors to disclose the network class assumed in disaggregation demos; reject Ethernet-only sites as 'Dynamo disagg ready' without math.
Security and privacy
What data, permissions and controls need testing?
Multi-server KV transfer widens the path carrying intermediate attention state; ensure encrypted fabrics and tenant isolation on NIXL paths.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Checklist culture (verify KV events, measure transfer throughput) matters more than model-tuning skills for first Dynamo pilots.
Procurement
What should contracts, pricing and exit terms secure?
Ask for measured KV transfer GB/s on the proposed fabric as a scored RFP item when disaggregated serving is proposed.
Operating model
Which teams own the service once it runs?
Operationalize the author's checklist: verify event publishing, measure same-prefix TTFT drop, and compute KV-bytes/link before any disagg go-live.
Publication history
- 2026-11-06NVIDIA · Issue 626 resources
Stable resource ID: ttlab-dynamo-hands-on-kv-disagg-link-bound-20261007