Why DeepSeek Stores KV Caches on SSDs to Accelerate Inference 26x While a SGLang Bug Leaked 155 Customer Sessions on Cloudflare
DeepSeek’s 100 PiB 3FS architecture serves 56% of tokens from NVMe SSDs, slashing 128K TTFT from 13s to 500ms, while Cloudflare’s postmortem reveals the tenant-isolation perils of disaggregated inference.
Published: October 29, 2026 | Read time: ~10 min | Watch Video (Japanese)
Executive Summary & Core Architectural Findings
The dogma of AI infrastructure long held that key-value (KV) caches—intermediate attention representations calculated during prefill—must remain strictly in expensive GPU memory (HBM). Conventional wisdom argued reading KV caches from solid-state drives (SSDs) would be far slower than having GPU Tensor Cores recompute attention from scratch.
In late October 2026, two landmark primary technical disclosures dismantled this assumption while exposing the architectural risks of modern inference systems.

First, DeepSeek open-sourced its large-scale cloud inference framework DeepInfer (powering DeepSeek V4.1 Flash) alongside technical documentation detailing its proprietary distributed storage engine, 3FS (Fire-Flyer File System). Operating across an approximately 100-pebibyte (PiB) dedicated NVMe SSD cluster, DeepSeek achieved radical operational metrics:
- Processing 155 trillion tokens per day on a single cluster.
- Generating $38 per GPU-hour in revenue even during off-peak valley pricing.
- Serving 56% of all production input tokens directly from NVMe SSDs rather than GPU HBM.
- Slashing time-to-first-token (TTFT) on 128K-token prompts by 26x, dropping from 13 seconds to 500 milliseconds.
- Lowering API pricing on cached tokens to one-fiftieth (50x less) of uncached prompt tokens.

Second, on October 26, 2026, Cloudflare engineers Kevin Flansburg and Michelle Chen published a postmortem detailing a critical multi-tenant isolation failure in their Workers AI hosting of @cf/zai-org/glm-5.3. Running the open-source SGLang engine in a prefill-decode (P/D) disaggregated topology via Mooncake RDMA, a mismatch between announced context limits (1,048,576 tokens) and actual KV pool capacity (681,280 tokens) caused 155 client responses to return un-overwritten KV cache pages containing confidential prompt and completion data from prior customers.
Together, these disclosures illustrate a historic inflection: LLM inference engines have evolved from basic matrix calculators into distributed virtual memory operating systems managing hierarchical tiers across HBM, host DRAM, and NVMe fabrics.

1. Why HBM Alone Breaks: MLA 93.3% Compression and the Physical Crossover

1-1. The Prefill Bottleneck in Enterprise AI
LLM inference operates in two distinct phases:
- Prefill Phase: Ingests prompts and calculates Key and Value activation vectors across all Transformer layers. This phase is compute-bound and dictates TTFT.
- Decode Phase: Generates output tokens autoregressively, one token at a time. This phase is memory-bandwidth bound, dictated by the speed of reading weights and cached states from HBM.
In multi-turn chat and agentic workflows, large portions of prompts (system prompts, tool schemas, repository trees) remain identical. Reusing cached KV representations skips costly prefill computation.
However, physical GPU memory is strictly bounded. As Baseten CTO Amir Haghighat observed, even clusters with eight NVIDIA H200 GPUs (1,128 GB aggregate HBM3e) quickly exhaust memory when hosting FP8 models under concurrent long-context demands.
1-2. The MHA Dilemma: Recomputation vs Disk Latency
Under standard Multi-Head Attention (MHA), storing individual Key and Value vectors for every head produces tens of gigabytes of KV data per request.
Historically, reading tens of gigabytes across PCIe and NVMe took several seconds—slower than recomputing the prefill directly on GPU Tensor Cores. Consequently, inference engines evicted old KV caches from HBM, forcing complete recomputation on subsequent turns.
1-3. How DeepSeek Reversed the Physics: MLA and FP4 Quantization
DeepSeek reversed this equation by combining Multi-head Latent Attention (MLA) with low-precision FP4 quantization:
- MLA Latent Projection: Instead of storing separate Key and Value vectors per head, MLA compresses them into a single low-dimensional latent vector per token. Keys and Values are dynamically projected during attention execution.
- 93.3% Capacity Reduction: MLA slashes KV cache memory footprint by 93.3% (a 15x reduction) compared to conventional standard attention architectures.
- DeepInfer FP4 Quantization: DeepSeek V4.1 Flash applies aggressive 4-bit quantization:
- Main KV: 16-element blocks in
e4m3(NVFP4 equivalent). - Indexer KV: 32-element blocks in
ue8m0, MXFP4. - MoE Routed Weights: 32-element blocks in
ue8m0, MXFP4.
- Main KV: 16-element blocks in
Compressing KV caches by over 15x reduced disk transfer volume proportionally. For the first time, reading compressed KV caches from NVMe SSDs became significantly faster than recomputing prefill on Tensor Cores.
2. Inside the 100 PiB 3FS Cluster and DeepInfer’s 3-Tier Cache Topology

Achieving multi-gigabyte-per-second throughput required purpose-built hardware and distributed file systems. DeepSeek engineered 3FS (Fire-Flyer File System) to back DeepInfer.
2-1. Storage Cluster Specifications
DeepSeek’s public architecture documentation outlines precise quantitative cluster specifications:
| Component / Parameter | DeepSeek V4.1 Flash Production Specification | Architectural Rationale |
|---|---|---|
| System Architecture | E/P/D Disaggregation (Vision Encoder / Prefill / Decode) + Dedicated 3FS Nodes | Decouples prefill compute from decode throughput |
| Parallelism Strategy | Prefill: EP2-8 / DP2-8; Decode: EP64-128 / DP64-128 (Wide-EP) |
Wide-EP minimizes memory traffic across MoE experts |
| Storage Node Fabric | 20x 30.73 TB NVMe SSDs per node (~614.6 TB raw/node); >40 GB/s read, >5 GB/s write | Aggregates bandwidth across thousands of NVMe drives |
| GPU to Storage Ratio | ~14.4 TB logical SSD capacity per online GPU (~100 PiB cluster total) | Stores days of global conversational context |
| Erasure Coding & Tail Latency | 10+2 Erasure Coding (EC); speculative fetch querying 11 nodes, reconstructing on first 10 | Mitigates network tail latency jitter |
| Cache Time-To-Live (TTL) | 3 days (72h+) for AI agents; 7 days for interactive chat | Chat exhibiting long-tail re-use curves |
2-2. Dedicated 3FS Nodes vs Local Server SSDs
Architects often ask why DeepSeek built a centralized 3FS cluster rather than caching KV pages on local server NVMe drives:
- Load Balancing vs Prefix Affinity: Caching locally on compute nodes forces prefix-matching requests to route to specific nodes, causing hot-spot congestion. With shared 3FS storage, any Prefill node can access any cached prefix, allowing load balancers to route purely on real-time GPU availability.
- Order-of-Magnitude Fault Isolation: Telemetry revealed that high-load GPU compute nodes fail more than an order of magnitude more frequently than storage nodes. Decoupling compute from storage ensures that when a GPU node crashes, 100 PiB of cached context remains intact.
2-3. GPUDirect RDMA Layered Swap-In and SWA Bounded Replay
DeepInfer organizes KV caches across three functional tiers:
- Tier 1 (GPU HBM): In-flight block KV and working state tensors.
- Tier 2 (Dedicated 3FS SSDs): Paged block KV (Main KV and Indexer K).
- Tier 3 (Host DRAM & 400G VLAN): Sliding Window Attention (SWA) cache. In V4.1, only 20 layers x 134 tokens x 528 bytes = ~1.41 MB per sequence is retained in host DRAM.
Transfer overhead is hidden through a pipelined hierarchy:
- Hash Matching: Entrypoint nodes match hash chains on arrival to identify cached token lengths.
- Asynchronous Retrieval: Daemons stream chunks from 3FS to host shared memory.
- GPUDirect RDMA (GDR) Layered Swap-In: Moves data from host memory to GPU HBM over RDMA, bypassing standard CUDA memory copies and freeing GPU SM cores.
Crucially, DeepInfer introduces SWA Bounded Replay. Reconstructing missing sliding-window attention typically requires recomputing up to 2,560 trailing tokens. DeepInfer truncates the attention window to the most recent 128 tokens, converting replay overhead into a constant cost and nearly halving total prefill computation.
3. Production Blind Spots: The Prefix-Match Trap and Cloudflare’s 155-Session Leak

3-1. The Exact-Match Trap in RAG and Agent Pipelines
The fundamental operational constraint of SSD-backed KV caching is that caching requires an exact match starting from token index zero.
In autoregressive causal attention, token representations depend strictly on preceding tokens. If an application developer prepends a dynamic timestamp (Current Time: 2026-10-29 20:15:03), a session ID, or dynamic RAG retrieval chunks at the beginning of a prompt, the cache hit rate collapses to 0%.
A prompt containing 100,000 tokens of static documentation loses all cache benefits, increasing TTFT from 500ms to 13 seconds (a 26x latency penalty) and raising token costs by 50x. Developers must design prompt templates with static prefixes positioned first and dynamic variables appended strictly at the tail.
3-2. Dissecting Cloudflare’s 155-Session Data Exposure
The risks inherent in disaggregated KV architectures materialized in Cloudflare’s security incident on @cf/zai-org/glm-5.3 between August 27 and September 9, 2026.

The Incident Timeline
- On September 8, 2026, security researcher Gadi Evron (Knostic) reported via HackerOne that oversized prompts to GLM-5.3 returned text from unrelated user sessions.
- Cloudflare reproduced the issue across separate test accounts within 25 minutes, offlined the model within 40 minutes, and patched configurations promptly.
- Across two weeks, 155 client requests exceeding 681,280 tokens received responses contaminated with fragments of other customers’ prompts and completions.
Root Causes: The Seven-Stage Failure Cascade
The incident stemmed from two intersecting defects in Cloudflare’s SGLang deployment:
- Capacity Misconfiguration: GLM-5.3’s public context limit was set at 1,048,576 tokens. However, SGLang calculates KV pool capacity dynamically based on remaining VRAM after loading model weights. Cloudflare’s Prefill nodes possessed memory for only 681,280 tokens. Pre-deployment integration tests checked only for HTTP 200 streaming status codes without validating output coherence on oversized inputs.
- SGLang Rejection Defect (Issue #30233 / #36988): To maximize throughput, inference engines do not zero-fill GPU memory pages upon deallocation; new requests overwrite prior data. When a prompt exceeded 681,280 tokens, a 7-stage bug chain triggered:
Cloudflare SGLang 7-Stage Leak Mechanism:
[1] Router -> Decode Nodes : Reserves 8 KV pages (P0 to P7) based on claimed length.
[2] Prefill Scheduler : Detects capacity overflow; triggers abort and replaces prompt with 1 dummy token.
Crucially, abort reason stored in `to_finish`, leaving `req.finished() == False`.
[3] Prefill GPU : Executes forward pass ONLY on the single dummy token.
[4] Mooncake RDMA : Transfers ONLY Page P0 to Decode, signalling completion.
[5] Decode Queue : Inspects `req.finished()` (still False); accepts request as valid.
[6] Decode Attention : Reads all 8 reserved pages (P0 to P7). P0 contains the dummy token,
but P1 to P7 contain un-overwritten KV data from prior customer requests.
[7] Response to Client : Emits HTTP 200 OK containing coherent, contaminated text from prior tenants.
4. Architectural Shift: Inference Engines as Distributed Virtual Memory Operating Systems

4-1. Replaying OS Kernel History in GPU Clusters
Inference engines have become distributed virtual memory operating systems orchestrating page tables across HBM, host DRAM, and remote NVMe fabrics.
The bugs Cloudflare encountered mirror classic OS kernel vulnerabilities:
- Page Lifetime & Use-After-Free: Concurrent token generation, host offloading, and abort signals referencing the same KV page caused active pages to be prematurely freed and reassigned (resolved in SGLang PRs
#24226,#30899,#30937). - Cross-Node RDMA Races: When clients disconnect while Prefill streams RDMA packets to Decode nodes, in-flight packets cannot be halted. Immediate deallocation caused late RDMA packets to overwrite newly assigned pages of subsequent requests (resolved via Deferred KV Release in PR
#41023).
4-2. Co-Design: Algorithms Dictate Storage Economics
DeepSeek’s ability to generate $38 per GPU-hour while cutting cache prices by 50x was achieved through holistic co-design:
- Model mathematics (MLA 93.3% compression + FP4 quantization) minimized transferred bytes.
- Algorithmic innovations (SWA Bounded Replay) capped recomputation to 128 tokens.
- Storage engineering (3FS 10+2 EC + GPUDirect RDMA) saturated PCIe fabrics to overlap compute with I/O.
Organizations separating model research from infrastructure operations cannot compete with this level of holistic optimization.

4-3. Integrity by Construction: The Future of Multi-Tenant Security
The Cloudflare incident proved that even if a model provider enforces strict Zero Data Retention (ZDR), bugs in KV memory management can leak raw customer data across tenant boundaries.
In response, Cloudflare and the SGLang community established new defensive paradigms:
- Page-Level Ownership Tags: Physical KV pages are stamped with generational ownership tags upon allocation. Decode nodes verify ownership tags before reading memory blocks during attention passes. If a mismatched tag is detected, the individual request fails safely, introducing less than 1% throughput overhead.
- Formal Verification of RDMA Protocols: Applying formal methods (such as those used in Cloudflare’s Topaz DNS engine) to mathematically prove the absence of race conditions in Mooncake RDMA state transitions.
5. Implementation Roadmap: Three Mandatory Production Actions

Engineering teams deploying LLM inference pipelines must adopt three immediate practices:
Action 1: Re-Architect Prompts for Strict Prefix Caching
- Implementation: Structure all prompts to place invariant elements first: system instructions, tool schemas, and static reference context. Position dynamic variables—session tokens, timestamps, user inputs, and dynamic RAG snippets—strictly at the end of the prompt template.
- Impact: Guarantees cache hits from token zero, reducing 128K TTFT from 13 seconds to 500ms and cutting input token costs up to 50x.
Action 2: Upgrade SGLang to v0.5.22+ and Enforce Deployment Verification
- Implementation: Teams operating disaggregated inference engines must upgrade to SGLang v0.5.22 or later and incorporate five mandatory CI checks:
- Compare startup logs to verify actual KV pool token capacity meets or exceeds published model context limits.
- Enable Deferred KV Release (
#41023) across Prefill and Decode roles. - Restrict Mooncake transfer batch sizes within timeout windows (
#32758). - Enable
--disaggregation-enable-kv-checksumand canary text validations to catch uninitialized memory reads. - Prevent immediate aborts on load balancers during client disconnects until prefill completes.
Action 3: Mandate KV Tenant Isolation Proofs in Provider Procurement
- Implementation: When procuring external inference APIs for confidential workloads, expand security audits beyond static ZDR policies. Require vendors to document how KV caches in DRAM and SSD storage are isolated, encrypted, and validated across tenant boundaries, verifying the presence of generational page ownership tags or zero-fill sanitization.
Primary Sources and References
- Daily Dose of Data Science (Avi Chawla): Why DeepSeek stores its KV cache on disk
- DeepSeek Official GitHub: DeepSeek V4.1 Flash Inference Deployment Q&A (DeepInfer)
- Cloudflare Security Blog: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang
- Latent Space Podcast: Everything you need to run Mission Critical Inference (ft. DeepSeek v3 + SGLang, Amir Haghighat & Yineng Zhang)