{"resourceId":"ttlab-dynamo-hands-on-kv-disagg-link-bound-20261007","versions":[{"version":"external-224aa983b64cdbc5414d01036d0ec9819b6d25967f3c59739e70e42cf591706b","resource":{"id":"ttlab-dynamo-hands-on-kv-disagg-link-bound-20261007","title":"TT Lab hands-on: Dynamo KV routing works; disaggregated serving >20× slower across two servers on 1GbE-class link","organization":"TT Lab (Toss ML/AI Infra series)","sector":"Independent AI infrastructure research","geography":"Author lab (single 8GB-class GPU + second consumer GPU; 1GbE-class inter-server)","publishedAt":"October 7, 2026","publicationDate":"2026-10-07","eventDate":null,"sourceName":"TT Lab Blog","sourceLabel":"Independent technical blog","sourceUrl":"https://ttlab.hopto.org/blog/toss-ml-infra-24-nvidia-dynamo-hands-on?lang=en","evidenceClass":"independent-research","outcomeClass":"cautionary","topics":["infrastructure","developers-agents","operating-model"],"finding":"Draft hands-on (Dynamo 1.6.0.dev, Qwen3-0.6B) finds KV-aware routing reliably steers repeated prefixes (up to 60/60 hits vs ~35–43 round-robin) but warns of silent degradation when KV events are missing; disaggregated serving was 4.7–7.2× slower on one shared GPU and more than 20× slower across two servers where TTFT matched KV bytes divided by ~104 MB/s link throughput. NVLink/RDMA benefits remain unmeasured.","sledRelevance":"Many SLED inference pilots still sit on enterprise Ethernet, not RDMA AI fabrics. This measurement shows disaggregated prefill/decode can be link-bound and harmful in that setting, while KV routing can still help—if event publishing is verified. Use as a pre-flight checklist beside NVIDIA Dynamo claims.","evidence":"Author tables: cross-server 1,500-word TTFT ~1.7 s vs standalone tens of ms; 3,000-word ~3.4–3.5 s; transfer ~351.75 MB at ~104 MB/s. KV event-disabled mode fell to random hit rates with only a log warning. Status marked Draft 2026-10-07; limitations section lists unmeasured RDMA/NVLink.","architectureImplications":"Estimate added TTFT ≈ (tokens × KV bytes/token) / measured transfer throughput before enabling disaggregation. Prefer KV routing pilots with event-health checks over disaggregation on slow links.","governanceImplications":"Require vendors to disclose the network class assumed in disaggregation demos; reject Ethernet-only sites as 'Dynamo disagg ready' without math.","securityPrivacyImplications":"Multi-server KV transfer widens the path carrying intermediate attention state; ensure encrypted fabrics and tenant isolation on NIXL paths.","caveats":"Draft post; tiny 0.6B model; synthetic prefixes; concurrency limited; dev build may differ from GA. Worst-case 1GbE-class link by design. Not peer-reviewed.","streamIds":["nvidia"],"roles":{"sales":"Use TT Lab as a practical brake on disaggregated-serving enthusiasm in Ethernet-heavy campuses and cities. Their hands-on saw KV routing work for repeated prefixes, but splitting prefill/decode across two servers on a 1GbE-class link made TTFT more than twenty times slower—matching bytes over link rate. Ask for the customer's east-west bandwidth before anyone demos disaggregation. Keep the tone constructive: routing may still be worth a pilot.","engineering":"Adopt the operational checklist: confirm workers emit KV events, measure hit-rate via same-prefix TTFT, and compute KV transfer time from bytes/token and iperf-class throughput before enabling disagg. Skip single-GPU 'disaggregation' tests as misleading. Treat RDMA/NVLink gains as unproven until measured on site.","delivery":"Structure pilots as routing-first. Gate any disaggregated serving change on a written transfer-time estimate and a measured link test. Document draft-status limitations when citing this source to stakeholders."},"roleDetails":{"sales":{"customerProblem":"Teams copy Dynamo disaggregation patterns without checking whether their network can move KV cache in time.","stakeholders":["Network architecture","Inference platform","CIO for AI factory pilots"],"discoveryQuestions":["What is measured bandwidth between potential prefill and decode nodes?","Are you proposing disaggregation or only KV-aware routing?","How will you detect silent KV-event failures?"],"valueHypothesis":"Routing-only pilots on reuse-heavy traffic may pay off; disaggregation should wait for fabric proof.","potentialEngagement":"Fabric-and-routing readiness checklist workshop before Dynamo feature enablement.","unsupportedClaims":["That 0.6B results scale linearly to 70B+","That RDMA would have shown gains here (unmeasured)","Peer-reviewed status"]},"engineering":{"fit":"Strong cautionary fit for Ethernet campus inference; partial positive for KV routing.","architectureIntegration":"Dynamo router modes; NIXL KV transfer paths; avoid disagg until link math clears.","prerequisites":["Ability to capture frontend KV-event warnings","Measured inter-node throughput","Same-prefix TTFT probe"],"constraints":["Draft + dev build","Tiny model / synthetic prefixes","RDMA/NVLink unmeasured"],"security":"Encrypt and isolate cross-node KV transfer; review multi-server blast radius.","proofOfValue":["Hit-rate improvement with events enabled","Disagg TTFT estimate within agreed budget before enable"]},"delivery":{"work":"Routing pilot with event-health checks; disagg only if transfer estimate passes.","dependencies":["Network team measurement support","Non-production Dynamo namespace","Stakeholder agreement on draft-source limits"],"ownership":"Platform engineering owns Dynamo flags; networking owns fabric evidence.","skillsAdoption":"Teach KV bytes/token estimation and log warning triage.","governanceCheckpoints":["Approve disaggregated serving enablement","Review silent-failure monitoring"],"acceptanceCriteria":["Proposal: no disagg enable without measured ≥X GB/s path","Proposal: alert on kv_event_publisher_disabled warnings"],"risks":["Stakeholders dismiss draft as non-authoritative","Teams still enable disagg from vendor diagrams","Dev/GA flag drift"]}},"retrievedAt":"2026-11-07T04:01:50Z","enrichedAt":"2026-11-07T04:01:50Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Checklist culture (verify KV events, measure transfer throughput) matters more than model-tuning skills for first Dynamo pilots.","procurementImplications":"Ask for measured KV transfer GB/s on the proposed fabric as a scored RFP item when disaggregated serving is proposed.","operatingModelImplications":"Operationalize the author's checklist: verify event publishing, measure same-prefix TTFT drop, and compute KV-bytes/link before any disagg go-live.","sourceVerification":{"openedUrl":"https://ttlab.hopto.org/blog/toss-ml-infra-24-nvidia-dynamo-hands-on?lang=en","referenceExcerpt":"Splitting across two servers was more than 20 times slower than standalone serving.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}