Memory and Storage for SI, Part 5: Context Memory

A model serving a long conversation keeps a record of every token it has already read. That record is the key-value (KV) cacheA KV cache stores the key and value vectors a language model computes for each token, so later tokens reuse them instead of recomputing them. It grows with context length., and on a long conversation it can outgrow GPU memory. NVIDIA’s Dynamo inference software moves the cache out of GPU memory to CPU RAM, local SSDs, or remote network storage, and on January 5, 2026, NVIDIA announced a storage platform built for that cache and calls that tier context memory.

Part 4 showed the cache spilling from GPU memory to system memory to flash and coming back as small random reads. This part asks where the cache goes once it leaves the GPU, and what each stop costs in room and in time. How SI Knows Anything covers what the cache holds and why attention needs it.

How big the cache gets

Attention reads a key and a value for every earlier token at every layer, so the cache stores both. NVIDIA’s inference guide writes the size per token as 2 × layers × (heads × values per head) × bytes per value, where the 2 counts keys and values. Its own example is Llama 2 7B at 16-bit precision, where one sequence of 4,096 tokens across 32 layers holds about 2GB.

Each model’s shape sets its cost per token. Kwon and colleagues’ PagedAttention paper works it out for OPT-13B as 2 × 5,120 × 40 × 2 bytes, which is 800KB per token, so one request at OPT’s 2,048-token limit can take 1.6GB.

Newer models shrink that by sharing keys and values among heads. Llama 3 uses grouped query attention (GQA) with 8 key-value heads to cut the cache during decoding, and Llama 3 405B has 128 query heads over 126 layers, with a width of 16,384. Split across the 128 heads, that is 128 values per head. Put through NVIDIA’s formula with the 8 shared heads, that is 2 × 126 × 8 × 128 × 2 = 516,096 bytes, about 516KB per token. A conversation of 128,000 tokens, close to the model’s 128K context window, holds about 66GB. Without the shared heads it would be 16 times larger.

NVIDIA’s Rubin GPU has 288GB of HBM4, enough in NVIDIA’s account for larger context windows without KV cache offload. At 66GB each, four of those long conversations fit in it (288 / 66 = 4.4), with no room for a fifth, before any of the model’s weights are counted. The cache grows linearly with batch size and sequence length, so a server that holds more conversations, or longer ones, runs out of HBM first.

Keeping it or computing it again

The cache is built during prefill. That first pass over a prompt computes keys and values for every token in it. When GPU memory runs out, NVIDIA names three ways out, which are to evict part of the cache and pay for recomputation, cap the prompt length, or add GPUs. Offloading is a fourth. In long sessions and multi-turn conversations it keeps the shared prefix of the prompt, so the next turn skips that part of prefill and the first token comes back sooner.

In a test NVIDIA reported with Vast Data, a DGX H100 system running Qwen3-32B with a 130K-token prompt loaded its precomputed cache from storage instead of recomputing it, which shortened the time to first token (TTFT). No figure for the saving was published.

Serving systems also waste the memory they have. Profiling existing systems, Kwon’s team found that only 20.4% to 38.2% of KV cache memory held actual token states, lost to room reserved for the longest answer a request might give and to fragmentation. Their vLLM engine stores the cache in fixed blocks the way an operating system pages memory, and it serves popular models at 2 to 4x the throughput of FasterTransformer and Orca at the same latency.

Reuse across requests pays more. Its authors report that, run with vLLM, LMCache reaches up to 15x the throughput on multi-round question answering and document analysis by storing caches and loading them back. Mooncake serves Moonshot AI’s Kimi assistant and is built around the cache, and its authors write that Kimi handles 75% more requests under real workloads with it.

Four tiers, G1 to G4

Dynamo’s KV Block Manager (KVBM) names the places a cache block can sit, and its design document numbers them G1 to G4. Other serving software uses the same ladder under other names: LMCache stores caches in CPU memory, local disk, remote disk, and Redis, and Mooncake pools the CPU, DRAM, and SSD resources of the GPU cluster into one cache.

G1 is GPU memory, the HBM where attention reads the cache. Attention reads from no other tier, so every block further down has to come back up before the model can use it. A Rubin GPU’s 288GB is the size of this tier.

G2 is host memory, a pool of pinned CPU memory that takes blocks offloaded from the GPU, sends them back, and offloads blocks on to disk. NVIDIA’s Vera CPU platforms carry up to 1.5TB of LPDDR5X per socket, several times what sits on one GPU.

G3 is local NVMe SSDs in the same server. Kioxia is pitching its CM9 drive for this job, in StorageReview’s report on Kioxia’s storage for SI: a 25.6TB E3.S model built on triple-level cell (TLC) flash and rated at 3 drive writes per day (DWPD), aimed at KV cache capacity in inference systems.

G4 is remote or cloud storage, which KVBM treats as an opaque blob store reached through NIXL.

Memory attached over a fabric is another option, outside KVBM’s list. In Astera Labs’ own figures, its Leo X-Series memory controllers, paired with its Scorpio X-Series switches, put memory on the GPUs’ scale-up fabric for long-context, KV-cache-heavy work, with up to 62% faster TTFT and up to 22% more tokens per second, against a baseline the release does not name. Flash behind a DRAM front tier on CXLCXL runs over the same physical layer as PCIe but adds cache-coherent memory access, so it moves cache lines rather than blocks of storage. is the other route, and Part 2 covers it.

How a block moves between tiers

KVBM moves the cache in blocks, each holding a fixed number of tokens for every layer, and the tiers form a chain. The GPU pool offloads to host memory, host memory offloads to the local disk pool, and a block comes back to the GPU either from host memory or straight from disk. The disk-to-GPU path skips a copy through host memory. Writes to a local disk go through NIXL, NVIDIA’s transfer library, as ordinary POSIX file writes, and over GPUDirect StorageGPUDirect Storage is NVIDIA’s path that moves data straight between an NVMe drive and GPU memory, skipping a copy through the CPU’s memory. when it is available, and the G4 tier is reached through NIXL as well.

Storage makers have measured the paths. In a Vast Data test NVIDIA reported, the GPUDirect Storage plugin in Dynamo moved the cache at 35GB/s to a single NVIDIA H100 GPU, enough to keep that GPU busy. WEKA’s lab test on a DGX system with eight H100 GPUs read the cache from its storage at up to 270GB/s across the eight. Both are vendor tests of their own systems.

Those figures are bandwidth, and bandwidth is only half of it. When cache data comes back as scattered random reads, StorageReview reports that each read is well under a kilobyte, so the number of reads per second a drive can serve sets the pace before its GB/s rating does. That is the problem Part 4 follows into the drives built for it.

Context memory storage: a tier of its own

At CES in January 2026 NVIDIA gave the shared tier its own product line. Its release argues that a KV cache cannot be kept on GPUs long term without becoming a bottleneck for real-time inference, and it introduces the NVIDIA Inference Context Memory Storage Platform, run by the BlueField-4 data processing unit (DPU). NVIDIA says the platform shares context across nodes at high speed, raises tokens per second by up to 5x, and gives up to 5x greater power efficiency than traditional storage. Storage makers building on BlueField-4 were to have products in the second half of 2026.

NVIDIA’s BlueField-4 STX rack gives the hardware a rack of its own and serves as a shared storage layer for KV cache data across a cluster, and its DOCA Memos software handles the cache in the storage processor and is said to raise inference throughput by up to 5x over general-purpose storage. NVIDIA’s FMS 2026 post names a context tier built on STX called NVIDIA CMX Context Memory Storage.

Read all three 5x figures against the baselines NVIDIA names, which are traditional storage for the platform and general-purpose storage for STX. They say what a purpose-built tier gains over a general one. They don’t say how CMX compares with a server that keeps the same cache in its own host memory or on its own SSDs, and none of the sources cited here makes that comparison.

What to watch

The first thing to check is hardware on sale. NVIDIA’s date for BlueField-4 platforms from storage makers was the second half of 2026, and a CMX tier measured in a customer’s cluster, with its TTFT and its power, is what would turn the 5x claims into a result.

The second is the pull between the two ends of the ladder. Rubin’s larger HBM keeps more conversations on the GPU without offload, while the context tier exists because some caches still don’t fit. Which one a deployment leans on depends on how long its contexts run and how often they come back, and the arithmetic in the first section gives the sizes to start from. My read is that long agent sessions keep the shared tier busy because at 66GB per conversation, only a few fit in HBM at once, and each one that returns needs its cache back from somewhere.

That flash has to come from somewhere. The series turns next to the drives built for capacity rather than speed.