Key fact: Ollama: q8_0 uses about half the memory of f16, q4_0 about a quarter; requires Flash Attention.
How we got it: Ollama FAQ, checked 14 October 2026.
Checked:
KV Cache Quantization in Ollama and llama.cpp
By LLM Hardware team · Methodology ·
Key facts
- Savings
- q8_0 about 1/2, q4_0 about 1/4 of f16
- Qwen3-32B 32K
- 8.6 GB f16, 4.3 GB q8_0
- Checked
- 2026-10-13
Checked 2026-10-13. Author: LLM Hardware team. This site does not store or show Amazon prices. Check current price on Amazon.
Citeable facts
Key fact: llama.cpp sets cache types with --cache-type-k and --cache-type-v.
How we got it: llama-server README, checked 14 October 2026.
Checked:
Key fact: Qwen3-32B uses about 8.6 GB of f16 KV cache at 32K tokens.
How we got it: 2 x 64 x 8 x 128 x 2 bytes x 32,768 tokens from config.json.
Checked:
Methodology — parameter math, quantization bytes, and the source list.
The KV cache stores attention keys and values for every token in the context. At long context it can rival the model weights in size. Both Ollama and llama.cpp can store it at lower precision to save memory.
How big the KV cache gets
KV cache bytes = 2 x layers x KV heads x head size x bytes per value x tokens. Layer and head counts come from each model config.
| Model | Context | f16 | q8_0 (about 1/2) | q4_0 (about 1/4) |
|---|---|---|---|---|
| Qwen3-32B (64 layers, 8 KV heads) | 32K | 8.6 GB | 4.3 GB | 2.1 GB |
| Qwen3-32B | 128K | 34.4 GB | 17.2 GB | 8.6 GB |
| Qwen3-30B-A3B (48 layers, 4 KV heads) | 32K | 3.2 GB | 1.6 GB | 0.8 GB |
| Qwen3-30B-A3B | 128K | 12.9 GB | 6.4 GB | 3.2 GB |
Example: Qwen3-32B at Q4 is 21.2 GB by our VRAM formula. Add a 32K f16 cache (8.6 GB) and it needs about 30 GB, which misses a 24 GB GPU. With a q8_0 cache it needs about 25.5 GB; with q4_0 about 23.3 GB, which fits 24 GB at the edge.
Ollama
The Ollama FAQ documents OLLAMA_KV_CACHE_TYPE. f16 is the default, q8_0 uses about half the memory with a very small loss in precision, and q4_0 about a quarter with a loss that can be more noticeable at long context. It needs Flash Attention (OLLAMA_FLASH_ATTENTION=1 forces it on) and applies to every model the server loads.
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve
llama.cpp
The llama-server README lists -ctk, --cache-type-k and -ctv, --cache-type-v, with values including f16, q8_0 and q4_0. Turn on Flash Attention with -fa when quantizing the V cache.
llama-server -m model.gguf -c 32768 -fa on -ctk q8_0 -ctv q8_0
When to leave the cache at f16
- Short chats: at 4K to 8K tokens the cache is small and there is little to save.
- Long-context work where accuracy matters most: q4_0 can lose more at long context, per Ollama. Try q8_0 first, or buy more memory. See best LLMs for 32 GB VRAM.
- If the weights themselves do not fit, cache quantization will not save you; use a smaller weight quant or MoE offloading.
Hardware
A 24 GB card is where cache quantization matters most for 27-32B models. Check current price on Amazon.
ASIN B0BG94PS2F. Amazon Associates link. The live price is on that page, not locked in here.
ASIN B0DS2X13PH. Amazon Associates link. The live price is on that page, not locked in here.
Sources
- Ollama FAQ: Flash Attention and K/V cache quantization
- llama.cpp llama-server README (cache types, --cpu-moe, --n-cpu-moe)
- Qwen3-32B config.json
- Qwen3-30B-A3B model card (30.5B total, 3.3B active)
- LLMHardware methodology: the VRAM formula
Questions
Does KV cache quantization reduce quality?
Ollama says q8_0 usually has no noticeable impact and q4_0 has a small to medium loss that can be more noticeable at long context.
How do I enable it in Ollama?
Set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 before starting ollama serve. It applies to all models.
A correction or a hardware question goes to the contact page.