kvrun vs self-hosting vLLM on your own GPUs
For many teams the real alternative to a managed inference service is not another vendor. It is “we’ll run vLLM ourselves.” That is a reasonable plan. vLLM is a capable, widely used serving engine, and owning the stack gives you control. This page sets out what self-hosting really costs, what you would have to build for encryption, isolation and erasure, and when self-hosting is the right call.
What self-hosting gives you
- Full control of the model, the serving configuration and the hardware.
- No provider in the path. Your requests stay on machines you run.
- Predictable spend once the GPUs are bought or reserved, if utilisation stays high.
Total cost of ownership
The cost of self-hosting is more than GPUs. Three lines matter most, and teams often underestimate the last two.
| Cost | What it includes | How teams miscount it |
|---|---|---|
| GPUs | Purchase or reserved hire, plus networking, storage and power or colocation | Counted at list price, with no spare capacity for failover or peaks |
| Engineering time | Setting up serving, scaling, upgrades, model rollouts, benchmarking and incident fixes | Treated as a one-off project, when it is ongoing |
| On-call | Someone responsible at 3 a.m. when a node fails or latency spikes | Assumed to be covered by existing staff, with no rota or runbooks |
Utilisation decides the economics. A GPU that sits idle at 20% is expensive per token however cheap the hardware looks on paper. A useful test: write down the GPU count you need at peak, add headroom for one node failing, then price the engineering and on-call hours each month. Compare that total with managed hours at your real volume, not your forecast.
What you would have to build for governance
vLLM is a serving engine. It is not a governance layer, and the controls that buyers ask for are not in the box. If you self-host, you will own each of these:
| Control | What you build | Why it is hard |
|---|---|---|
| Encryption of the KV cache | Sealing cached blocks at rest and in every offload tier, with key management | The cache is in GPU memory and moves between tiers, so encryption must hold on the hot path |
| Tenant isolation | Separate keys, cache and placement per customer, and no prefix sharing across tenants | Prefix caching is a performance feature, so turning it off or partitioning it costs throughput |
| Erasure | A way to delete a session's data from every tier, and evidence that it happened | Overwriting does not reliably remove every copy, so you need key-based erasure and a certificate |
| Audit evidence | An append-only log of allocation, access and deletion, exportable for an auditor | Logs must be complete and tamper-evident, which is a design task, not a dashboard |
| Residency | Policy that keeps compute and storage in the UK | Every replica, cache tier and backup needs checking, not just the primary region |
Each of these is solvable. The question is whether your team should spend months building and maintaining them, rather than the product your customers are paying for.
When self-hosting wins
Self-hosting is the right call in some cases, and it is worth saying so plainly:
- Large, stable, high-utilisation workloads on one or two models, where GPUs stay busy and the cost per token is hard to beat.
- An in-house MLOps team that already runs production GPU infrastructure, with on-call and runbooks in place.
- Strict requirements to own the hardware, for example air-gapped environments or on-premises rules.
- Custom serving work that a managed service does not support, such as unusual kernels or architectures.
When managed governed inference wins
- You want open models on dedicated GPUs without building the platform underneath them.
- You need encryption, isolation, erasure and audit evidence on day one, with a certificate you can show a regulator.
- Your volume is growing or uneven, and you do not want to carry idle GPUs.
- Your team is small, and the engineering hours are better spent on your product.
kvrun runs open models on dedicated GPUs in London behind a governed KV cache, with an OpenAI-compatible API. You keep the model choice and the endpoint shape, and you do not run the serving stack, the isolation layer or the erasure process yourself.
How to decide
- Write down your peak GPU count, headroom and utilisation for the next 12 months.
- Price GPUs, engineering hours and on-call for self-hosting, honestly.
- List the governance controls your buyers and regulators require, and cost each one as a build.
- Run a pilot on both, using your own prompts, and compare cost and time to a production-ready system.
Questions
- Is vLLM secure enough for regulated data?
- vLLM is a serving engine. Security for regulated data depends on the controls around it: encryption, isolation, erasure, residency and logging. Those are yours to build and prove if you self-host.
- Can I move from self-hosting to managed later?
- Yes, if you keep the API shape and model choice portable. An OpenAI-compatible endpoint makes that move simpler.
- What if I only need a few GPU hours a month?
- Managed inference is usually cheaper at low volume, because you are not paying for idle capacity or the engineering to keep it running.