KV cache tiering: how to fit 4× the context on one GPU

KV cache tiering keeps the attention state a model needs right now in GPU memory, and moves the rest to cheaper, larger tiers: host memory, local NVMe and object storage. Because the KV cache, not the weights, is usually what runs out of room on long-context work, tiering lets one GPU hold far more context than its VRAM alone. kvrun's early internal benchmarks show up to 4× the context per 80 GB GPU. This post explains where that figure comes from and where it stops applying.

Why the KV cache fills the GPU

For every token in context, the model stores a key and a value vector in every layer. The size per token is:

bytes per token = 2 (K and V) × layers × KV heads × head dimension × bytes per value

Take Llama 3.1 8B: 32 layers, 8 KV heads, a head dimension of 128, stored in 16-bit precision. That is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB, per token. A single 128k-token conversation therefore needs about 16 GiB of KV cache, on top of the 16 GB of weights. Ten users with long contexts at once would need 160 GiB, twice an 80 GB GPU, before counting anything else.

Weights are a fixed cost. The KV cache grows with every token and every concurrent user, so it is what decides how many long conversations a GPU can hold.

The tiers

Each step down is roughly an order of magnitude larger and slower than the one above. Tiering works because most context is not needed every millisecond: a user reading a reply, a conversation paused overnight, a 200-page contract several people ask about through the day.

Reload or recompute?

When evicted context is needed again, there are two options: load it back, or recompute it with a fresh prefill. Prefill is compute-bound and its cost rises with prompt length, while reload is bandwidth-bound and rises with the bytes moved. For short prefixes, recomputing is often faster than fetching from a slow tier. For long ones, loading back from host memory or NVMe usually wins comfortably, and that is where tiering pays off.

A good tiering system makes this choice per block, prefetches context it expects to need, such as the history of a user who has just started typing, and overlaps transfers with computation so the GPU is not left waiting.

Where the 4× comes from, and its limits

The multiplier is not a property of the hardware alone. It depends on how your traffic uses context:

• Many sessions, few active at once. This is the best case. Idle sessions live in the lower tiers and come back quickly, so the GPU holds many times more total context than it could alone.
• Shared long prefixes. A long document or system prompt cached once in host memory serves every request that starts with it, within the same tenant.
• One enormous context, attended to continuously. This is the worst case. Every decoding step needs the whole cache, so offloaded blocks must stream back for every token, and bandwidth becomes the limit. Tiering still lets it run, but it does not make it fast.

Treat 4× as what we have measured on typical multi-user, long-context workloads, not a guarantee. Measure it on your own traffic, looking at time to first token on resumed sessions as well as throughput.

Tiering and security

Tiering has a governance cost that is easy to miss. Every tier is another copy of your users' prompts, and a block written to NVMe or object storage may outlive the request that created it. In most open-source offload setups, those copies sit unencrypted on disk.

On kvrun, each block is sealed with AES-256-GCM under a per-tenant key before it leaves the GPU, and stays sealed on every tier. Erasing a session means destroying its key, which makes every copy on every tier unreadable at once, with a certificate to show for it. Encryption, policy checks and audit writes add under 3% to time to first token in early internal benchmarks. See the governed KV cache docs.

What it means for sizing

Without tiering, you size a GPU for all the context any user might hold. With tiering, you size it for the context being actively decoded at peak, and let disk hold the rest. That is why kvrun's deployment wizard sizes hardware from concurrent users rather than context length, and why a longer context window costs storage rather than a bigger GPU. See Deploy your first model.