{"resourceId":"campi-dynamo-kv-router-saturation-503-20260627","versions":[{"version":"external-224aa983b64cdbc5414d01036d0ec9819b6d25967f3c59739e70e42cf591706b","resource":{"id":"campi-dynamo-kv-router-saturation-503-20260627","title":"Independent test: Dynamo KV-router under load returns 503s (~1 in 7) to keep latency low—trade-off, not pure win","organization":"Michele Campi (independent)","sector":"Independent AI infrastructure research","geography":"Lab measurement on 8×A100-SXM4-40GB (author setup)","publishedAt":"June 27, 2026","publicationDate":"2026-06-27","eventDate":null,"sourceName":"Michele Campi technical blog","sourceLabel":"Independent technical blog","sourceUrl":"https://michelecampi.github.io/observability/systems-engineering/llm-inference/2026/06/27/dynamo-kv-router-saturation.html","evidenceClass":"independent-research","outcomeClass":"cautionary","topics":["infrastructure","developers-agents","operating-model"],"finding":"Independent A/B on Dynamo with Mooncake-trace load shows the KV-aware router is not simply faster: near saturation (N=2 workers on the author's 8×A100 box) it shed ~14.3% of requests with HTTP 503 ResourceExhausted while round-robin completed 100%, trading rejects for better latency among accepted calls; the gap vanished with more workers.","sledRelevance":"Direct counterweight to NVIDIA Dynamo KV-routing marketing for shared campus or municipal inference gateways that cannot silently drop citizen or researcher requests. SLED operators must decide whether latency for accepted traffic or completion rate is the governing SLO before enabling kv router mode.","evidence":"Author reports at N=2: kv-router completed 2571/3000 (429 failures, 14.3%) vs round-robin 3000/3000; failures are explicit 503 ResourceExhausted. At N=4 and N=8 both arms completed 100%. Workload is Mooncake production-style trace with prefix sharing; Dynamo v1.2.0 pinned.","architectureImplications":"Default KV-router behavior under overload is admission control via 503, not graceful queueing. Sites need client retry/backoff, capacity headroom, or different router settings before exposing public endpoints.","governanceImplications":"Publish whether 503 shedding is acceptable for each service class (internal research vs public digital services).","securityPrivacyImplications":"Error responses and retries can amplify load; ensure rate limits and auth still apply on retry paths.","caveats":"Single-author lab on one 8×A100 box; Dynamo v1.2.0 may differ from later releases. Benefit profile is saturation-specific. Not a SLED production deployment.","streamIds":["nvidia"],"roles":{"sales":"Bring this independent result when customers hear only Dynamo 'faster TTFT' claims. Under load, Campi's test found the KV-router shed about one in seven requests with 503s to protect latency for the rest—while round-robin accepted everything and got slower. Ask which matters more for their service: p50 latency or completion rate. Do not weaponize the post as 'Dynamo is broken'; position it as an operating-policy discovery.","engineering":"Reproduce saturation A/B on the site's GPU count and trace before enabling --router-mode kv on shared gateways. Instrument completed vs failed requests, not only TTFT. Plan client retry/jitter and consider keeping round-robin for public endpoints until headroom is proven. Pin and record Dynamo version because behavior may change.","delivery":"Include a saturation test gate in any Dynamo routing pilot: fixed arrival schedule, report completion %, 503 rate, and TTFT for both arms. Acceptance must state a max reject rate. Own the decision with the service product owner, not only platform SRE."},"roleDetails":{"sales":{"customerProblem":"Inference teams may enable KV routing from vendor docs without knowing overload returns 503s.","stakeholders":["Platform SRE","Digital services product owner","Research computing","Risk/compliance"],"discoveryQuestions":["Is dropping requests with 503 acceptable for this API?","What completion-rate SLO exists today under peak load?","Do clients already retry idempotently?"],"valueHypothesis":"Explicit overload policy plus measured headroom may let KV routing help reuse-heavy traffic without surprising user-visible failures.","potentialEngagement":"Saturation test design and SLO workshop for inference gateway routing modes.","unsupportedClaims":["That later Dynamo versions behave identically","That campus GPUs will match 8×A100 lab ratios","That round-robin is always preferable"]},"engineering":{"fit":"Critical review input for any site considering Dynamo KV routing on multi-tenant GPUs.","architectureIntegration":"Compare --router-mode kv vs round-robin; add metrics for HTTP 503/ResourceExhausted.","prerequisites":["Load generator with fixed schedule","Per-request success/failure logs","Version-pinned Dynamo deployment"],"constraints":["Effect strongest near saturation","Author used defaults; tuned queues may differ","Single-box CUDA_VISIBLE_DEVICES carving"],"security":"Ensure 503 paths do not leak internals; retries must re-auth.","proofOfValue":["Completion rate and TTFT curves at N matching production","Documented reject-rate threshold with product owner sign-off"]},"delivery":{"work":"Run three-seed A/B saturation test; publish a go/no-go for kv mode.","dependencies":["Representative trace or synthetic prefix overlap","Maintenance window on pilot pool","Client owners to interpret 503s"],"ownership":"Platform SRE runs test; service owner sets reject-rate policy.","skillsAdoption":"Teach reading AIPerf/completion metrics alongside latency.","governanceCheckpoints":["Approve production kv routing","Approve public vs internal exposure"],"acceptanceCriteria":["Proposal: reject rate ≤ agreed % at target load","Proposal: runbook for 503 storms published"],"risks":["Silent user harm if only latency dashboards watched","Version drift invalidates lab lessons","Over-generalizing one author's box"]}},"retrievedAt":"2026-11-07T04:01:24Z","enrichedAt":"2026-11-07T04:01:24Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Operators need load-test skills and clear runbooks for 503 storms; citizen-facing services need UX for retries.","procurementImplications":"Require vendors demo saturation behavior (completion rate and error codes), not only best-case TTFT slides.","operatingModelImplications":"Add saturation drills to inference acceptance; define max acceptable reject rate before enabling KV routing in production.","sourceVerification":{"openedUrl":"https://michelecampi.github.io/observability/systems-engineering/llm-inference/2026/06/27/dynamo-kv-router-saturation.html","referenceExcerpt":"it returns 503s to about one in seven callers — and that is precisely how it keeps latency low","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}