{"resourceId":"red-hat-developer-tiered-kv-cache-offloading-openshift-ai-3-5-h100","versions":[{"version":"external-597745d67d1d46550c3819042618a9e05d4241700b5040e856b88f5d5a207f5e","resource":{"id":"red-hat-developer-tiered-kv-cache-offloading-openshift-ai-3-5-h100","title":"Red Hat Developer: Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5","organization":"Red Hat","sector":"Cross-sector AI infrastructure (applicable to state data centers and university research computing)","geography":"Global (tests on Red Hat OpenShift and IBM Cloud VPC)","publishedAt":"November 2, 2026","publicationDate":"2026-11-02","eventDate":null,"sourceName":"Red Hat Developer","sourceLabel":"Vendor engineering benchmark","sourceUrl":"https://developers.redhat.com/articles/2026/11/02/benchmarking-tiered-kv-cache-offloading-h100-gpus-openshift-ai-3-5","evidenceClass":"vendor-claim","outcomeClass":"mixed","topics":["infrastructure","developers-agents"],"finding":"Vendor-reported benchmark: on OpenShift AI 3.5 (vLLM 0.24.0, llm-d 0.8.0, OpenShift 4.22.2, 8xH100 nodes) replaying an agentic trace averaging about 50,600 input tokens, tiered KV cache offloading raised Nemotron-Ultra-253B single-replica throughput from 0.13 to 0.76-0.77 req/s (up to 5.8x) and cut mean TTFT from 59.7 s to 9.6 s. CPU offload alone gave 3.3x for Nemotron; for Qwen3.6-35B-A3B, whose working set fit the CPU tier, CPU offload captured about 90% of the gain. Results were mixed elsewhere: Gemma 4 31B gains were constrained by GPU memory and its 2-replica mean TTFT regressed from 3.3 s to 4.4 s; at concurrency 128, p95 TTFT exceeded 400 s in every configuration. CephFS results used single-copy (size 1) replication. Red Hat recommends a customer proof of concept using FIO p99 latency and a restore-versus-recompute crossover calculation.","sledRelevance":"Relevant to universities and state shared-services data centers planning on-premises long-context or agentic inference on OpenShift AI, where GPU memory is the scarce resource. It provides a reproducible PoC method rather than a SLED deployment; no public-sector workload was tested. Why now: published after the archive's last coverage of OpenShift AI 3.5 inference features and alongside the 5 Nov Red Hat/NVIDIA AI Factory inference reference design, and it was not previously archived.","evidence":"Vendor-run benchmark with stated hardware, software versions, models, trace (SemiAnalysis Weka AgentX trace, 30-minute runs, seed 42), FIO storage method and per-configuration results. No independent replication; single trace; no cost-per-request figures; CephFS minimal replication; preemption contribution not isolated (authors state an ablation would be required).","architectureImplications":"CPU DRAM is the first offload tier and a mandatory staging layer for secondary tiers; local NVMe suits single-node deployments; CephFS (ODF) suits multi-replica deployments with shared prefixes. Object storage (RGW/S3) showed p99 up to 1.13 s for 5 MB blocks. Sizing must consider multi-GB/s read bandwidth on the shared tier for large models.","governanceImplications":"Performance claims should be validated on the customer's own trace before procurement decisions; the vendor explicitly frames thresholds as illustrative, not universal.","securityPrivacyImplications":"KV cache blocks persisted to NVMe or shared CephFS contain derived prompt state; a shared tier extends the data footprint of prompts across nodes and needs access control and retention review. Interpretation: Red Hat's CVE-2026-94624 (Important) describes a vLLM DoS when TieringOffloadingSpec uses a peer-to-peer secondary tier; the benchmark used filesystem tiers, not P2P.","caveats":"Vendor self-benchmark; not peer reviewed or independently reproduced. Results depend on model, block size, concurrency and storage tail latency. CephFS used size-1 replication, so durability differs from production defaults. No energy or cost measurements. Not a SLED deployment.","streamIds":["red-hat"],"roles":{"sales":"Sales framing: this is a capacity-efficiency story for customers already constrained on GPUs, not a general speed claim. The headline 5.8x applies to one large model whose working set exceeded the CPU tier; another model gained about 2x and one regressed at two replicas. Use it with university research computing or a state data center to qualify long-context or agentic workloads, then propose a bounded PoC on their own trace rather than quoting the vendor multiplier.","engineering":"Engineering framing: the article supplies a usable method: compute KV block size, run FIO at that block size for p99, derive the restore-versus-recompute crossover, then replay a representative trace across no-offload, CPU, CPU+NVMe and CPU+CephFS. Check GPU memory headroom first, size CPU offload deliberately, and treat CephFS size-1 results as non-production. Avoid peer-to-peer secondary tiers unless network exposure is tightly restricted, given CVE-2026-94624.","delivery":"Delivery framing: offloading adds storage tiers that must be operated, monitored and secured. Plan for NVMe or ODF capacity, cache-hit and TTFT dashboards, and data-retention rules for cached prompt state. Run the four-step PoC with agreed latency SLOs, document results per model, and revisit when models or vLLM versions change. Do not promise production gains until the customer's trace and replication settings have been tested."},"roleDetails":{"sales":{"customerProblem":"Long-context RAG and agent workloads exhaust GPU memory, forcing recomputation and long time-to-first-token on scarce GPUs.","stakeholders":["Research computing director","State data center / shared services infrastructure lead","AI platform owner","Storage team lead"],"discoveryQuestions":["What share of inference requests exceed roughly 2,000-50,000 input tokens?","Are GPU nodes at memory capacity, and is TTFT an SLO today?","What local NVMe or Ceph/ODF capacity already exists in the OpenShift estate?","How many replicas serve the same prefixes or documents?"],"valueHypothesis":"For long-context, prefix-reusing workloads, tiered KV offloading may increase throughput per GPU and reduce TTFT, deferring GPU purchases, subject to PoC validation.","potentialEngagement":"A two-to-four-week KV offload PoC on the customer's trace following Red Hat's FIO and crossover procedure, with a written tier recommendation.","unsupportedClaims":["A guaranteed 5.8x throughput gain","GPU cost savings in dollars","Gains for short-context workloads","Production durability of the size-1 CephFS configuration"]},"engineering":{"fit":"Fits OpenShift AI 3.5 deployments serving long-context or agentic workloads with prefix reuse on GPU-memory-constrained nodes.","architectureIntegration":"vLLM OffloadingConnector with TieringOffloadingSpec: GPU HBM, CPU DRAM tier, then local NVMe or CephFS (ODF) filesystem tier via llm-d serving on OpenShift AI.","prerequisites":["OpenShift AI 3.5 with vLLM/llm-d serving","Sufficient pinned CPU memory per replica","Local NVMe or ODF CephFS storage class","Representative inference trace"],"constraints":["Gains depend on working set vs. CPU tier size","CephFS p99 108-451 ms limits short-context benefit","GPU memory headroom can cap gains","Results tested only on NVIDIA H100"],"security":"Cached KV blocks on disk or shared storage hold derived prompt state; restrict access, set retention, and avoid P2P tiers on reachable networks (CVE-2026-94624).","proofOfValue":["Compute KV block size and run FIO p99 at that size","Calculate crossover context length per tier","Replay customer trace across four configurations and compare TTFT, throughput and cache-hit rate","Test with production replication settings, not size 1"]},"delivery":{"work":"Provision offload tiers, configure the connector, run the PoC matrix, and operationalize dashboards for TTFT, cache hits and storage tail latency.","dependencies":["Storage team for NVMe/ODF provisioning","Platform team for vLLM configuration","Application owner for representative traces"],"ownership":"AI platform / inference operations team with storage operations support.","skillsAdoption":"Requires vLLM tuning, storage performance testing (FIO) and Ceph/ODF operations skills.","governanceCheckpoints":["Data-retention review for cached prompt state","Security review of tier network exposure","Change approval before enabling shared tiers in production"],"acceptanceCriteria":["Proposed: measured TTFT and throughput improvement on the customer trace meets an agreed threshold","Proposed: storage p99 below the computed crossover for target context lengths","Proposed: no TTFT regression at planned replica count"],"risks":["Gains smaller than vendor figures on customer workloads","Additional storage cost and operational burden","Cache data exposure on shared storage"]}},"retrievedAt":"2026-11-07T22:08:30-06:00","enrichedAt":"2026-11-07T22:12:38-06:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Adds storage and inference-tuning skill demands to small university or state platform teams.","procurementImplications":"May shift spend from additional GPUs toward CPU memory and NVMe or ODF storage; procurement should require PoC evidence on the buyer's workload.","operatingModelImplications":"Inference operations must monitor cache-hit rate and storage tail latency alongside GPU metrics.","sourceVerification":{"openedUrl":"https://developers.redhat.com/articles/2026/11/02/benchmarking-tiered-kv-cache-offloading-h100-gpus-openshift-ai-3-5","referenceExcerpt":"tiered KV cache offloading improved throughput by up to 5.8x and reduced mean time to first token (TTFT) from 60 s to 10 s","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}