Is prefix caching safe? Tenant isolation in shared LLM serving
Prefix caching is safe within one tenant and unsafe across tenants, unless the cache is partitioned. When a serving engine reuses cached attention state for requests that start with the same tokens, a cache hit makes the response measurably faster. Several published papers show that an attacker sharing the cache can use that timing difference to learn what other users sent, sometimes token by token. Partitioning the cache by tenant closes the channel; encryption alone does not.
This extends our earlier piece, Your KV cache holds your prompts, with the research and the mitigations in detail.
What prefix caching does
To answer a prompt, a model first computes attention keys and values for every token, a step called prefill. That state, the KV cache, is then reused while the answer is generated. Prefill is expensive on long prompts, and many prompts start the same way: a shared system prompt, a few-shot template, a document several users ask about.
Prefix caching keeps the KV cache from earlier requests and reuses it when a new request begins with the same tokens. vLLM does this by hashing fixed-size blocks of tokens, building on the paged memory design in the PagedAttention paper. SGLang's RadixAttention keeps cached prefixes in a radix tree. Both can cut time to first token sharply on repeated prefixes, which is why the feature is on in most production deployments.
Why it leaks
A cache hit skips work, so a hit is faster than a miss. If two tenants share one cache, tenant A can send a guess and time the response. A fast response means someone has already sent a prompt with that prefix.
On its own, that confirms a guess. Repeated, it becomes extraction. An attacker who knows or guesses the start of a prompt, for example a template, can try each likely next token, keep the one that hits, and repeat. Block-based caches leak at block granularity, which slows the attack but does not stop it.
What the research shows
- The Early Bird Catches the Leak (Song et al.) demonstrated timing side channels in LLM serving, including prefix and semantic caches, and showed prompts being recovered from shared caches. It proposes sharing only in large chunks of tokens to enlarge the attacker's search space.
- I Know What You Asked (Wu et al., NDSS 2025) introduced PromptPeek, an attack on KV-cache sharing in multi-tenant serving with frameworks such as vLLM and SGLang, reconstructing other users' prompts token by token. It considers attackers with different levels of prior knowledge, including one who knows the template and needs only a few key tokens.
- Auditing Prompt Caching in Language Model APIs (Gu et al., ICML 2025) tested real commercial APIs with statistical timing audits. It detected prompt caching in 8 of 17 providers, and global cache sharing across users in 7 of them. The authors disclosed their results to the affected providers in October 2024.
The last point matters most for buyers. This is not a lab curiosity on a modified engine. Cross-user cache sharing has been found in production APIs, and several providers have since changed their behaviour. You cannot assume yours was not one of them, or has not changed since.
Mitigations, and how well they work
| Mitigation | What it does | Does it close the channel? |
|---|---|---|
| Turn prefix caching off | No reuse, so no hit/miss signal | Yes, but you lose the speed-up on every repeated prefix |
| Partition the cache per tenant | Only requests from the same tenant can hit each other's blocks | Yes, across tenants. Users inside one tenant can still probe each other |
| Dedicated deployment | No other customer's traffic shares the engine | Yes, across customers. Same caveat inside the deployment |
| Share only in large chunks | A hit requires a long matching prefix | Partly. It raises the cost of the attack but does not remove it |
| Add timing noise | Blurs the difference between hit and miss | Partly. Noise can be averaged away with enough queries, and it slows everyone |
| Selective sharing | Share only entries judged safe, for example SafeKV or CacheSolidarity | Promising research. Not yet a standard feature you can rely on |
| Encrypt the cache | Protects cached blocks in memory and on disk | No, on its own. Encryption hides the contents, not whether a hit happened |
Partitioning is the control that works without giving up the performance. vLLM supports it through a per-request cache salt: the salt is mixed into the hash of the first block, so requests with different salts never share blocks, even for identical prompts (vLLM prefix caching design). The same documentation warns that a non-cryptographic block hash raises the risk of collisions that could leak data between tenants, and recent versions default to SHA-256. If you run vLLM yourself, check both settings.
Encryption and partitioning do different jobs, and you need both. Encryption protects the contents of the cache from anyone with access to memory or disk. Partitioning removes the timing signal between tenants. Neither does the other's job.
How kvrun handles it
Every kvrun deployment is a dedicated GPU instance, so no other customer shares the serving engine or its cache. Inside a deployment, send an X-KV-Tenant header with your end user's ID and each user gets their own sealed cache and their own prefix-cache salt: two users sending byte-identical prompts do not share a block. Each block is sealed with AES-256-GCM under a per-tenant key, and allocation, access and erasure are written to an audit log. Details are in the governed KV cache docs.
Partitioning per user still allows reuse where it is safe: one user's repeated system prompt and conversation history hit the cache. What it gives up is reuse between users, which is exactly the reuse the attacks depend on.
Questions to ask your provider
- Is prefix or prompt caching shared between customers? Between users inside our organisation?
- If it is partitioned, what is the partition key, and can we set it per end user?
- What hash function identifies cached blocks?
- How long does a cached prefix live, and is it written anywhere other than GPU memory?
- Is cached context encrypted, and under whose key?
- Can you show us a log of cache access for our tenant?
Questions
- Is prefix caching safe for a single-tenant internal tool?
- Usually yes. If everyone sharing the cache is allowed to see everyone else's prompts, the timing channel reveals nothing they could not already see. Partition by user once different users' data must stay apart.
- Does turning on encryption stop prefix caching attacks?
- No. Encryption protects what is stored, but the attack only needs to know whether a cache hit happened. You need partitioning for that.
- Does partitioning make the model slower?
- Only for reuse between partitions, which is the unsafe kind. Reuse within a partition, such as a user's own conversation history, still hits the cache.