Run kvrun on your own machines

Self-hosting puts the kvrun inference stack on hardware you own. One binary brings up the same containers the managed platform runs on a cloud GPU: the GKV inference engine, the marking proxy every response passes through, and optionally a chat interface. Your prompts, completions and model weights never leave your machine.

This guide goes from an empty Linux server to a working endpoint, then covers the day-to-day work: updating, moving a licence to a new server, and what the licence check does and does not do.

curl -fsSL https://get.kvrun.com/install.sh | sh
kvrun preflight
kvrun activate XXXXX-XXXXX-XXXXX-XXXXX-XXXXX
kvrun serve qwen3.5-9b

Those four commands are the whole path. The rest of this page explains each one.

Managed or self-hosted?

Choose self-hosting when the data cannot leave your estate, when you already own GPUs, or when you need to run somewhere we cannot reach. Choose managed when you would rather not run servers at all. Where your workloads run compares every placement.

Before you start

You need:

• Linux on x86-64 or arm64.
• An NVIDIA GPU and its driver. nvidia-smi must work.
• Docker Engine with the Compose plugin, and the NVIDIA Container Toolkit, so containers can see the GPU.
• Disk space. The GKV engine image is about 11 GB to download and roughly three times that once unpacked. Model weights come on top of that (about 20 GB for a 9B model at full precision).
• A licence key. A free 30-day evaluation key is fine to start with; see Get a licence key.

kvrun will not install or reconfigure your driver or Docker for you. We do not change machines we do not own. kvrun preflight checks everything at once and lists every problem together, so you fix them in one pass rather than one failure at a time.

Network access

Serving itself needs no connection to us. These hosts are used for setup, licence checks and downloads:

If your servers have no internet access at all, you need an offline licence, which we arrange through the Enterprise tier.

Get a licence key

Licences are issued in the kvrun console under Self-hosting.

• Evaluation. Free for 30 days, counted from the day you activate the key rather than the day it is issued. One machine and one model at a time, for evaluation and non-production use only. You need a saved card to start one (it is not charged), and each card can start one evaluation. When it ends, its models stop.
• Team, Business and Enterprise. Commercial licences for production use. Each covers a set number of machines, shown on its card in the console. Enterprise adds unlimited machines, air-gapped installation and your own contract terms.

Your key appears on the Self-hosting page once the licence is issued, in the form XXXXX-XXXXX-XXXXX-XXXXX-XXXXX, together with the install command.

Install

curl -fsSL https://get.kvrun.com/install.sh | sh

The installer finds the right build for your CPU, downloads it, checks it against the published checksums, and installs kvrun to /usr/local/bin. It asks for sudo when it needs it. It also sets up an hourly timer that keeps the licence check current (see How the licence check behaves).

Then check the machine, and fix anything it reports before going on:

kvrun preflight

To install somewhere you can write without root, set a prefix:

curl -fsSL https://get.kvrun.com/install.sh | KVRUN_PREFIX=$HOME/.local sh

Activate the licence

kvrun activate XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

This verifies the key with the licensing service, records it on the machine, and takes one of the licence's machine seats. The first activation on a machine must be able to reach the licensing service. After that, the machine can run offline for up to 30 days between checks.

kvrun license shows what the machine is entitled to at any time: the tier, whether commercial use is permitted, the expiry date, and the machine's ID.

If every seat on the licence is already in use, activation is refused and tells you how to free one. See Move a licence to another machine.

Serve a model

kvrun serve qwen3.5-9b

The first run pulls the images and downloads the weights, so it takes a while. Later starts reuse both. When it returns, you get the endpoint and its key:

Serving qwen3.5-9b
Id         d1a2b3c4
Model      Qwen/Qwen3.5-9B
Engine     GKV kernel
Endpoint   http://127.0.0.1:8000/v1
API key    kvrun_local_…

The model keeps loading in the background after serve returns. Watch it with kvrun logs <id> --follow until it reports that it is ready.

Choosing a model

kvrun models lists every catalog name serve accepts. These are the same names the managed platform uses, such as qwen3.5-9b, and each one maps to its Hugging Face repository. You can also pass any Hugging Face repository directly as org/name:

kvrun serve meta-llama/Llama-3.1-8B-Instruct

For gated models, accept the model's licence on Hugging Face, then set your token when you serve: HF_TOKEN=hf_… kvrun serve <model>.

Which engine runs it

Where it can, kvrun serves with the GKV kernel, our own inference engine. It seals the KV cache under a key generated on your machine, which never leaves it. Where the kernel cannot serve a model, vLLM serves it instead, and serve tells you which engine it chose and why. Today vLLM is used for:

• models sharded across several GPUs,
• mixture-of-experts models,
• Hugging Face repositories outside the kvrun catalog.

Add --engine vllm to serve a model with vLLM regardless.

The first time a machine serves with the GKV kernel, it downloads the engine image (about 11 GB) from get.kvrun.com, checks it against its published checksum, and loads it into Docker. That happens once per machine.

Options

Endpoints listen on 127.0.0.1 by default, so nothing is reachable from your network until you say so. To publish one, pass --bind 0.0.0.0 and put it behind your own firewall or reverse proxy. Each model gets its own port, starting at 8000.

Call the endpoint

The endpoint speaks the OpenAI API, so any OpenAI client works. Point it at the endpoint, and use the API key serve printed:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $KVRUN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.5-9b", "messages": [{"role": "user", "content": "Hello"}]}'

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="kvrun_local_…")
reply = client.chat.completions.create(
    model="qwen3.5-9b",
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)

The key is stored in the deployment's folder on the machine and shown by kvrun list. It is never sent to us. There is more on SDKs and streaming in Call your endpoint from code.

AI-generated content marking

Every response passes through the marking proxy, so every reply, whether from the API or the chat interface, is marked as AI-generated in line with Article 50 of the EU AI Act. The marking arrives in an X-CP2E-Marking response header and a cp2e field in the response body, and it is signed with a key generated on your machine. Markings from your installation therefore verify against your key, not ours. The proxy is the same code the managed platform runs, so the markings are identical in form.

Day to day

Models keep running when you log out or the CLI exits: they are ordinary Docker containers, set to restart with the machine.

Update kvrun

sudo kvrun update

This downloads the newest release from get.kvrun.com, refuses to install it unless it matches the published checksum, starts the new binary once to make sure it runs on your machine, and only then swaps it in. Running models are not touched, so updating causes no downtime.

kvrun update --check tells you whether a newer version exists without installing it. kvrun also prints a one-line notice on everyday commands when one is available, and the console's Installations list marks any machine running an older version.

Move a licence to another machine

Each licence covers a set number of machines. To replace a server, free its seat first, then activate the new one.

If the old server is still running, run this on it:

kvrun release

It stops the server's models (they stay on disk) and gives the seat back straight away. Then, on the new server:

kvrun activate XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

If the old server is gone or unreachable, open Self-hosting in the console. Every machine holding a seat on your licence is listed under Installations, with a Release button beside it. Releasing frees the seat immediately, so the new server can activate straight away. If the old server ever comes back online, it finds out at its next licence check (within the hour), stops its models, and does not take the seat back.

A released server is not wiped. Running kvrun activate on it again takes a seat back if one is free, and kvrun start <id> restores its models as they were.

You do not need to cancel or re-issue the licence to move a seat, and the other machines on it are unaffected.

How the licence check behaves

The check is designed so that a licence problem never takes your production down.

• At start-up and hourly, the machine re-verifies its licence. The hourly check runs from the timer the installer sets up.
• If the licensing service cannot be reached, the machine keeps running on its last verified licence for up to 30 days.
• When a commercial licence expires or is revoked, or those 30 days run out, kvrun refuses to start new models and says so loudly on every command. Models that are already serving keep serving.
• Two cases stop running models: an evaluation that has ended, because it is nobody's production, and a machine you released yourself, because its seat now belongs to another machine.

Reinstalling kvrun does not use up another seat. A machine keeps its identity across reinstalls.

What leaves your machine

To check the licence and count machines, kvrun sends the licence key, a machine ID, the kvrun version, the hostname, how many models are running and how many GPUs the machine has. That is the complete list.

Never sent: prompts, completions, model weights, adapters, datasets, logs, user data, or your endpoints' API keys. The key that signs content markings and the key that seals the KV cache are both generated on your machine and stay on it.

Reference

Files on the machine

Environment variables

Uninstall

Release the machine's seat first, so the licence is free for another one:

kvrun release --yes
sudo systemctl disable --now kvrun-report.timer
sudo rm -rf /usr/local/bin/kvrun /usr/local/lib/kvrun /var/lib/kvrun

kvrun release stops the models. Their containers and images stay in Docker until you remove them with docker system prune if you want the space back.

Troubleshooting

Preflight says the machine is not ready

Fix each item it lists. The usual one is the NVIDIA Container Toolkit: install it, run sudo nvidia-ctk runtime configure --runtime=docker, then restart Docker.

Activate says every machine the licence covers is in use

Free a seat: run kvrun release on the machine you are replacing, or release it from the console's Installations list. See Move a licence to another machine.

Serve says a model is not in the catalog

Check the spelling with kvrun models, or pass the full Hugging Face repository as org/name.

The model never becomes ready

Run kvrun logs <id> --follow. A 401 from Hugging Face means the model is gated: accept its licence on Hugging Face, stop the deployment, and serve it again with HF_TOKEN set. A CUDA out-of-memory error means the model is too large for the GPU: try a smaller one.

The machine says it was released

Its seat was freed from the console or by kvrun release. Run kvrun activate <key> to take a seat back, if one is free.

Anything else

Run kvrun report, and send its output with the relevant kvrun logs output through our contact page.