Blog

Local LLM Mania

· 6 min read

Introduction

Translation note This article is an English translation of a Japanese article originally published on Zenn.

Once you start running LLMs locally, questions come up: why does this quantization make the model faster? Why does the same model run at a different speed when I switch runtimes?

To understand this behavior, I want to cover the following five concepts:

  1. Prefill / Decode
  2. KV Cache
  3. Quantization
  4. Runtime
  5. Drafter / Speculative Decoding


Prefill / Decode

LLM inference is broadly split into two phases: prefill and decode.

Prefill is the phase that processes the input prompt. Given a 4,000-token prompt, for example, the model first processes all 4,000 tokens together and computes the information it needs to generate the first token.

It then moves on to decode, generating new tokens one at a time based on the context so far.

sequenceDiagram participant U as Prompt participant M as LLM U->>M: 4,000 Tokens Note over M: Prefill M-->>U: Token 1 Note over M: Decode M-->>U: Token 2 M-->>U: Token 3 M-->>U: Token 4 M-->>U: ...

Because prefill can process many tokens at once, it maps onto large matrix operations and makes good use of the GPU's parallelism. Decode, on the other hand, is sequential: each new token is computed from the token generated just before it.

Because of this difference, prefill and decode also have different bottlenecks. Where you should look depends on whether you want to speed up prompt processing or improve tok/s during generation.



KV Cache

During decode, every time a new token is generated, attention refers back to all of the tokens before it.

Attention computes a key (K) and a value (V) for each token, and once the K/V for a past token has been computed, it does not change afterwards. The KV cache is the mechanism that stores these already-computed K/V values and reuses them.

sequenceDiagram participant M as Model participant C as KV Cache Note over M: Prefill M->>C: Store K/V for Token 1 M->>C: Store K/V for Token 2 M->>C: Store K/V for Token 3 Note over M: Decode C-->>M: K/V of past tokens M->>C: Append K/V for Token 4 C-->>M: K/V for Tokens 1–4 M->>C: Append K/V for Token 5

Without a KV cache, every decode step would have to recompute the K/V of all past tokens as well. With a KV cache, past results are reused, and only the K/V for the newly added token needs to be computed.

The trade-off is that the longer the context gets, the more K/V there is to store.

flowchart LR Context["Context Length"] -->|grows| KV["KV Cache"] KV -->|grows| VRAM["VRAM usage"]

So even when the model itself fits in VRAM, increasing the context length can still run you out of VRAM.

An easy way to think about the KV cache is as a mechanism that trades memory for less computation.



Quantization

In an LLM, the weights that make up the model account for most of the memory usage.

Holding the weights of a 70B model in FP16, for example, takes roughly 140 GB by simple arithmetic. Quantization is the technique of representing these weights with fewer bits.

flowchart LR FP16["FP16
16 bit / weight"] --> Q8["8-bit Quantization"] FP16 --> Q4["4-bit Quantization"] Q8 --> Small["Smaller weights"] Q4 --> Small

Quantization is not only a way to fit a model into less VRAM. During decode, a large amount of weight data is read from memory for every token generated, so smaller weights also mean less memory traffic.

flowchart LR Q["Fewer bits"] --> W["Smaller weights"] W --> T["Less memory traffic"] T --> D["Shorter decode time"] D --> TPS["Higher tok/s"]

The effect is especially large when decode is bound by memory bandwidth.

However, because quantization represents the original weights approximately, it introduces quantization error. The fewer bits you use, the more memory you save, but the more likely the model's accuracy and benchmark scores are to drop.

Lower bit widths are also not always faster.

If the runtime has no kernels that can process that quantization format efficiently, extra work such as dequantization can keep you from getting the speed you expected.



Runtime

A model file stores the weights and the model's architecture, but that alone is not enough to run inference.

The software that loads the model and actually performs the computation on the CPU or GPU is the runtime, also called the inference engine. For local LLMs, llama.cpp is an easy example to picture.

Here is where the runtime sits:

flowchart TB Model["Model / GGUF"] --> Runtime["Runtime
e.g. llama.cpp"] Runtime --> Memory["Memory management"] Runtime --> KV["KV Cache management"] Runtime --> Scheduler["Scheduling"] Runtime --> Backend["Backend"] Backend --> CPU["CPU"] Backend --> CUDA["CUDA"] Backend --> Metal["Metal"] CUDA --> Kernel["Kernel"] Metal --> Kernel CPU --> Kernel

The runtime is responsible for loading the weights, placing them on the CPU/GPU, GPU offload, managing the KV cache, matrix operations, attention, choosing kernels, and more.

That is why, even with the same model and the same quantization format, tok/s changes when the runtime, the backend, or the kernels it uses change.



Drafter / Speculative Decoding

In normal decode, each run of the target model generates one next token.

sequenceDiagram participant T as Target Model participant O as Output T->>O: Token 1 T->>O: Token 2 T->>O: Token 3 T->>O: Token 4

With a large model, each of these decode steps takes time.

So a smaller, faster model is used to predict the upcoming tokens, and the target model verifies those candidates together. The model that generates the candidates is called the drafter (draft model), and this inference method is called speculative decoding.

sequenceDiagram participant D as Drafter participant T as Target Model participant O as Output D->>D: Predict Tokens 1–4 D->>T: Draft Tokens Note over T: Verify together T-->>O: Commit accepted tokens D->>D: Predict the next tokens D->>T: Draft Tokens T-->>O: Commit accepted tokens

When the drafter's predictions closely match the target model, a single verification pass by the target model can advance several tokens.

When the match rate is low, on the other hand, many tokens are rejected, and the extra cost of running the drafter is no longer worth it.

flowchart LR Draft["Draft Tokens"] --> Verify{"Verify with the target model"} Verify -->|"High acceptance rate"| Fast["Advances several tokens"] Verify -->|"Low acceptance rate"| Slow["Barely advances"] Fast --> Gain["Easy to speed up"] Slow --> Loss["Drafter overhead grows"]

So when evaluating a drafter, what matters is not simply its model size but the combination of the drafter's own speed and its acceptance rate.



Original Japanese Article

This write-up is an English translation of my original Japanese article:

ローカルLLMマニア — Zenn

« All posts

Local LLM Mania 0%
12:00 PM