Local LLM Mania
Introduction
Once you start running LLMs locally, questions come up: why does this quantization make the model faster? Why does the same model run at a different speed when I switch runtimes?
To understand this behavior, I want to cover the following five concepts:
- Prefill / Decode
- KV Cache
- Quantization
- Runtime
- Drafter / Speculative Decoding
Prefill / Decode
LLM inference is broadly split into two phases: prefill and decode.
Prefill is the phase that processes the input prompt. Given a 4,000-token prompt, for example, the model first processes all 4,000 tokens together and computes the information it needs to generate the first token.
It then moves on to decode, generating new tokens one at a time based on the context so far.
Because prefill can process many tokens at once, it maps onto large matrix operations and makes good use of the GPU's parallelism. Decode, on the other hand, is sequential: each new token is computed from the token generated just before it.
Because of this difference, prefill and decode also have different bottlenecks. Where you should look depends on whether you want to speed up prompt processing or improve tok/s during generation.
KV Cache
During decode, every time a new token is generated, attention refers back to all of the tokens before it.
Attention computes a key (K) and a value (V) for each token, and once the K/V for a past token has been computed, it does not change afterwards. The KV cache is the mechanism that stores these already-computed K/V values and reuses them.
Without a KV cache, every decode step would have to recompute the K/V of all past tokens as well. With a KV cache, past results are reused, and only the K/V for the newly added token needs to be computed.
The trade-off is that the longer the context gets, the more K/V there is to store.
So even when the model itself fits in VRAM, increasing the context length can still run you out of VRAM.
An easy way to think about the KV cache is as a mechanism that trades memory for less computation.
Quantization
In an LLM, the weights that make up the model account for most of the memory usage.
Holding the weights of a 70B model in FP16, for example, takes roughly 140 GB by simple arithmetic. Quantization is the technique of representing these weights with fewer bits.
16 bit / weight"] --> Q8["8-bit Quantization"] FP16 --> Q4["4-bit Quantization"] Q8 --> Small["Smaller weights"] Q4 --> Small
Quantization is not only a way to fit a model into less VRAM. During decode, a large amount of weight data is read from memory for every token generated, so smaller weights also mean less memory traffic.
The effect is especially large when decode is bound by memory bandwidth.
However, because quantization represents the original weights approximately, it introduces quantization error. The fewer bits you use, the more memory you save, but the more likely the model's accuracy and benchmark scores are to drop.
Lower bit widths are also not always faster.
If the runtime has no kernels that can process that quantization format efficiently, extra work such as dequantization can keep you from getting the speed you expected.
Runtime
A model file stores the weights and the model's architecture, but that alone is not enough to run inference.
The software that loads the model and actually performs the computation on the CPU or GPU is the runtime, also called the inference engine. For local LLMs, llama.cpp is an easy example to picture.
Here is where the runtime sits:
e.g. llama.cpp"] Runtime --> Memory["Memory management"] Runtime --> KV["KV Cache management"] Runtime --> Scheduler["Scheduling"] Runtime --> Backend["Backend"] Backend --> CPU["CPU"] Backend --> CUDA["CUDA"] Backend --> Metal["Metal"] CUDA --> Kernel["Kernel"] Metal --> Kernel CPU --> Kernel
The runtime is responsible for loading the weights, placing them on the CPU/GPU, GPU offload, managing the KV cache, matrix operations, attention, choosing kernels, and more.
That is why, even with the same model and the same quantization format, tok/s changes when the runtime, the backend, or the kernels it uses change.
Drafter / Speculative Decoding
In normal decode, each run of the target model generates one next token.
With a large model, each of these decode steps takes time.
So a smaller, faster model is used to predict the upcoming tokens, and the target model verifies those candidates together. The model that generates the candidates is called the drafter (draft model), and this inference method is called speculative decoding.
When the drafter's predictions closely match the target model, a single verification pass by the target model can advance several tokens.
When the match rate is low, on the other hand, many tokens are rejected, and the extra cost of running the drafter is no longer worth it.
So when evaluating a drafter, what matters is not simply its model size but the combination of the drafter's own speed and its acceptance rate.
Original Japanese Article
This write-up is an English translation of my original Japanese article: