890 bytes per token: how DeepSeek shrinks the KV cache

When a model processes a long document, it stores the attention Key and Value for every token in the context. This intermediate result is the KV cache. It grows linearly with context length, determining how many long sessions a single card can run simultaneously. This is why long context costs often depend on memory rather than weights.

You can calculate the space a KV cache occupies using a public model. Take Qwen3-8B and its published configuration. At BF16 or FP16, where each value takes 2 bytes, each token corresponds to 147456 bytes of KV data, or about 147KB. The calculation is 2 for K and V, times 36 layers, times 8 KV heads, times 128 dimensions per head, times 2 bytes. This excludes extra overhead like cache management.

DeepSeek applied compression in V4.1 Flash. The official comparison shows that compared to the previous generation, HBM demand drops to a quarter and SSD demand to an eighth. Relative to the first-generation model, the KV cache is 437 times smaller. Combining official data with descriptions from various sources, V4.1 Flash uses about 890 bytes of KV cache per token.

Why focus on this number? Cache hits account for a large share of costs in agent-style tasks. A coding agent repeatedly reads code, looks up information, and runs tests, causing history to accumulate. When VRAM cannot hold it, the system clears part of the cache. The next round must then redo Prefill for the needed data, increasing the wait time for the first token.

The engineering solution is tiering. Data currently being generated stays in HBM. Data likely to be reused soon moves to CPU-side DDR memory. Data untouched for longer periods sinks to SSD or remote storage. vLLM's KV Offloading, for example, supports offloading cache blocks to CPU memory and configuring a secondary storage tier, with retrieval passing through the CPU-side cache layer.

So 890 bytes is more than just a number. It means the same VRAM can support more concurrent long sessions, and teams running their own inference services need fewer cards. For procurement and operations, KV cache size is now as important as model performance and belongs on the selection checklist.