LoRA vs QLoRA: which to use on private data

Use QLoRA by default when the model is large for your GPU, when you are still iterating on the dataset, or when cost matters most. Use LoRA when you have settled on the data and want the highest-fidelity adapter, and the 16-bit base model fits your hardware comfortably. On private data, the method matters less than what you do with the adapter afterwards, because it is a derivative of your data either way.

This is the decision. For the steps, see Fine-tune a model with LoRA or QLoRA. For the wider case for fine-tuning in the UK, see Fine-tuning open models on private data.

What each one is

LoRA (Hu et al., 2021) freezes the base model and trains a small pair of low-rank matrices alongside selected layers. The trained matrices, the adapter, are typically a fraction of a percent of the model's size. The base weights are held in 16-bit precision during training.

QLoRA (Dettmers et al., 2023) trains the same kind of adapter, but holds the frozen base model in 4-bit precision, using a 4-bit NormalFloat format, double quantisation of the scaling constants, and paged optimiser states. The paper fine-tuned a 65B model on a single 48 GB GPU and reported results matching 16-bit fine-tuning on its benchmarks.

The trade-offs

Those memory figures are for the frozen weights only. Activations, the adapter's gradients and optimiser state come on top, and grow with sequence length and batch size.

How to choose

1. Does the 16-bit base model fit your GPU with room for activations? If not, use QLoRA. It is the only way to tune that model on that hardware.
2. Are you still changing the dataset? Use QLoRA for cheap iterations, and decide later whether a final LoRA run is worth it.
3. Is the task precision-sensitive? Exact extraction, numbers, code or a strict output format can benefit from LoRA. Run both on a held-out set and compare.
4. Will you serve many adapters on one base model? Either works. The adapter's size and format are the same.

What changes on private data

Neither method keeps your training data in a readable form, but both produce weights learned from it. Small adapters trained for many epochs on small datasets can memorise examples, and a model can sometimes be prompted into reproducing them. For data protection purposes:

• Treat the adapter as derived from personal data if the training set contained any. Store it, control access to it and list it in your records of processing on that basis.
• Minimise first. Remove names, identifiers and free-text fields the task does not need before training. It is far easier than removing them from weights afterwards.
• Watch the epochs. More epochs on small data raise the risk of memorisation. Three is a sensible default; go higher only with a held-out set showing it helps.
• Plan for erasure. If someone exercises the right to erasure and their data was in the training set, the dependable answer is to retrain the adapter without it. Adapters are cheap to retrain, which is a real advantage over full fine-tuning.
• Test for leakage. Prompt the tuned model with the opening of training examples and check it does not complete them verbatim.

Where the training runs matters too. On kvrun, training runs on a GPU in London that shuts itself down when the job ends, and the adapter is written to your workspace storage, from where you can deploy it to its own governed endpoint.