KV cache at 890 bytes per token: how DeepSeek squeezed attention memory

The most striking figure in the DeepSeek-V4.1-Flash spec is not the parameter count. It is memory. The model card states that the global KV cache occupies 890 bytes per token, roughly a quarter of the size in DeepSeek-V4-Flash. The vendor’s release describes the same effect from an infrastructure angle: HBM demand falls to a quarter, and SSD demand to an eighth, compared with the previous generation.

This matters because agentic tasks force the model to read long contexts repeatedly. Cache hits account for a large share of the compute bill, yet the KV cache must still reside somewhere physical. When a context reaches hundreds of thousands of tokens, the cache stops being an implementation detail. It becomes the primary constraint on throughput and server cost.

Three mechanisms instead of one trick

Compression was achieved through several independent methods. The first is SWA Bounded Replay. Missing attention-window states are reconstructed by replaying only the last tokens of the window, eliminating the need to store them permanently on SSD. The persistent KV footprint drops to about an eighth of its pre-change size.

The second is Comprossed Sparse Attention 2 (CSA2). It assigns each attention layer to one of three modes: Full, Reindex, or Reuse. This allows layers to share the main KV and the indexer key, while sparse attention indices are reused. In the decoder, a hierarchical sparse indexer limits deeper indexing layers to a candidate pool built by the first layer in Full mode. Indexing cost no longer scales with context length.

The third is storing the main KV in FP4 format: values in E2M1 representation with one E4M3 scale per 16 channels. Only the combination of all three elements yields 890 bytes per token.

What it changes in practice

A smaller KV footprint means more parallel sessions and longer contexts fit on the same server, with less data moving between memory and disk. The vendor’s note explicitly promises lower serving costs and states that savings will be passed to users, alongside peak and off-peak pricing. DeepSeek describes this as “pushing the limits of KV cache compression,” a race not over who has more parameters, but over who maintains long-context memory more cheaply.

The Chinese portal IT之家 notes that on the deployment side, V4.1-Flash has been integrated into, among other places, a national supercomputer network. There, reducing HBM and SSD demand is a prerequisite for serving multiple users simultaneously.