The governed KV cache and its audit log
The KV cache is the attention keys and values a transformer keeps for every token it has processed, so each new token does not recompute the whole prompt. It is large, it grows with context, and it is a faithful working copy of what your users sent. kvrun treats it as governed data rather than disposable plumbing.
For why this matters at all, read governed inference. This guide is about what you see and control in the console.
Where cached context lives
The KV cache section of a deployment reports the tiers that deployment is using.
| Tier | What it is | How it is protected |
|---|---|---|
| GPU memory | The working set, on the accelerator | Reserved for your deployment alone |
| Memory tier | Host RAM on the same box, for blocks evicted from the GPU | Sealed per tenant with AES-256-GCM |
| Persistent tier | Your own workspace storage, so a restart does not start cold | Ciphertext only, under a key only that deployment holds |
Persisted blocks are written under the deployment's own folder in your storage, restored after restarts and rebuilds, and deleted with the deployment. Stateless deployments have the persistent tier switched off entirely, and the console says so and why.
The context window is bounded, 32,000 tokens by default unless the model sets its own, because the scratch space for decryption is sized for the longest prompt a deployment will see.
One deployment, many of your users
Send X-KV-Tenant: <user-id> and each of your end users gets their own sealed cache and their own prefix-cache salt. Two users sending byte-identical prompts do not share a cached block. Without the header, the deployment is a single tenant: fine for an internal tool, not for a product with customers in it.
The audit log
Every component of the cache records, per tenant, what it sealed, opened, reused, evicted, persisted and restored, along with any integrity failure. Records are hash-chained: each carries the hash of the one before it, so an edited, missing or reordered record is detected when the log is collected.
- The control plane collects the log off the instance every few minutes and verifies the chain. You can also collect it on demand.
- Segments are downloadable from the deployment page, and from Storage under the deployment's own folder.
- The log is kept after the deployment is deleted, because evidence that disappears with the thing it describes is not evidence.
The downloadable, hash-chained log is produced by the vLLM engine. A deployment on another engine still records cache statistics on the instance, but cannot produce the collectable log. The deployment page says which you have, so redeploy on vLLM if you need the paper trail.
Erasure
Deleting a session, a workspace or an account destroys the key that decrypts its data rather than chasing rows. Every copy on every tier becomes unreadable at once, including copies nobody can reach into any more, such as last night's backup. That is the deletion that satisfies a right-to-erasure request, and it is issued with a certificate.
What the evidence answers to
| Framework | Requirement | What you can show |
|---|---|---|
| UK GDPR Art. 17 | Right to erasure | An erasure certificate per session |
| UK GDPR Art. 25 | Data protection by design | Encryption and per-tenant isolation on by default |
| SOC 2 CC6.1 | Logical access | Per-tenant key isolation |
| SOC 2 CC7.2 | Monitoring | The continuous, hash-chained audit log |
| ISO 27001 | Secure disposal | Cryptographic erasure |
| EU AI Act | Evaluation records | The evaluation kept beside every fine-tune's weights |
kvrun produces evidence that maps to these controls. Certification of your organisation remains between you and your auditor.
Questions
- Does any of this slow the model down?
- Not noticeably. Early benchmarks show under 3% added time to first token. Measure your own workload: the figure moves with context length.
- Can I turn the cache off?
- Choose a stateless deployment and nothing is persisted between requests. The in-memory tiers still exist, because that is how a transformer serves a single request.
- Who can read the audit log?
- Anyone in your workspace with access to the deployment or to Storage. It records cache events, not prompt contents.
- What happens to the log when I delete the deployment?
- It is kept. The cached ciphertext and its key go; the record of what happened stays.