How Many Tokens Per Second Should You Expect from a Local LLM?
By LLM Hardware team · Methodology ·
Updated 14 November 2026 · LLM Hardware team
A local LLM's generation speed is capped by how fast the machine can read the model weights. A simple memory-bound ceiling is memory bandwidth divided by the bytes of weights read for each new token. At Q4_K_M this site uses 0.5 bytes per parameter, the same constant as the methodology page, so an 8B dense model reads 4.0 GB of weights per token. On an RTX 4090 (1008 GB/s, NVIDIA Ada whitepaper, Table 2) that ceiling is about 252 tok/s. Real speed is lower. A 70B dense model at the same quant reads 35 GB of weights per token, so the same card's ceiling is about 29 tok/s, and the model does not fit in 24 GB of VRAM (44 GB by our formula). The 12-row table below uses that same arithmetic on vendor-published bandwidth. This site does not measure hardware.
The memory-bound decode ceiling
The Roofline model of Williams, Waterman, and Patterson says attainable performance is the minimum of peak compute and peak memory bandwidth times operational intensity. A kernel that hits the slanted part of that roof is memory bound. Their peer-reviewed paper is in Communications of the ACM 52, no. 4 (2009). The official technical report of the same work, UCB/EECS-2008-134 (17 October 2008), states the bound as:
Attainable GFlops/sec = Min(Peak Floating Point Performance, Peak Memory Bandwidth x Operational Intensity)
Source: UCB/EECS-2008-134.pdf (HTTP 200, checked 14 November 2026). The ACM Digital Library landing page for the 2009 article returned HTTP 403 on that date, so the quotation is from the official technical report, not from a paid ACM PDF.
That paper is about floating-point kernels, not language models. The step from the paper to token generation is an application of the same bound, not a measurement in the paper. During autoregressive decode, each new token typically requires reading the model weights once. If that traffic dominates, a memory-bound ceiling is:
tok/s ceiling = memory bandwidth (GB/s) / (parameters in billions x bytes per parameter)
Assumptions used in the table:
- Dense models only in the four size columns. Every parameter is read once per generated token.
- Quantization is Q4_K_M at this site's published constant of 0.5 bytes per parameter (see methodology). An 8B model is 4.0 GB of weights per token, 14B is 7.0 GB, 32B is 16.0 GB, and 70B is 35.0 GB. This is the weights term only. It is not the VRAM formula, which multiplies by 1.2 and adds 2 GB for short-context runtime and a reserve.
- Bandwidth is the vendor-published figure linked in the table, in GB/s. Ceilings are rounded the same way as the site's
/can-i-runhelper (Math.round(bandwidth / (params x 0.5))insrc/lib/speed-estimate.ts). - A cell says "does not fit" when the site's Q4_K_M VRAM formula (params x 0.5 x 1.2 + 2) is larger than the card's VRAM, or, on Macs, larger than unified memory minus the site's 4 GB macOS reserve. The ceiling is hidden in those cells because the simple bound assumes the weights are read from the device memory whose bandwidth is listed. Offloading to system RAM is a different bound.
- The ceiling is an upper bound, not a speed you should expect. Real generation is lower: the memory system is not 100% efficient, the KV cache and other working data also move, and a batch-1 chat has overheads this formula does not model.
Worked example, RTX 4090 at 1008 GB/s: 1008 / 4.0 = 252 tok/s for an 8B Q4_K_M dense model; 1008 / 16.0 = 63 tok/s for 32B; 1008 / 35.0 = 28.8, which rounds to 29 tok/s for 70B. The 70B model still does not fit 24 GB (44 GB by the VRAM formula).
Computed Q4_K_M ceilings for 12 catalog machines
All 12 rows exist in data/hardware.json. Bandwidth figures are taken from the linked vendor page or whitepaper (checked 14 November 2026), not from a run on our hardware. The RTX 4080 row uses 716.8 GB/s from the Ada whitepaper Table 3; the catalog stores 716. For these four model sizes the rounded ceilings are the same either way. NVIDIA GeForce RTX 50-series figures are from the official compare table. Apple figures name the chip configuration, because M5 Max bandwidth is 460 GB/s on the 32-core GPU SKU and 614 GB/s on the 40-core GPU SKU.
There is no measured tok/s column. This page only lists a computed ceiling where the vendor bandwidth and the fit rule above are both known. Community benchmark repositories are not used here.
| Machine (catalog) | Vendor bandwidth | 8B Q4 | 14B Q4 | 32B Q4 | 70B Q4 |
|---|---|---|---|---|---|
| NVIDIA GeForce RTX 5070 (12 GB) | 672 GB/s | 168 | 96 | does not fit (21.2 GB) | does not fit (44 GB) |
| NVIDIA GeForce RTX 4080 (16 GB) | 716.8 GB/s | 179 | 102 | does not fit (21.2 GB) | does not fit (44 GB) |
| NVIDIA GeForce RTX 5080 (16 GB) | 960 GB/s | 240 | 137 | does not fit (21.2 GB) | does not fit (44 GB) |
| NVIDIA GeForce RTX 3090 (24 GB) | 936 GB/s | 234 | 134 | 59 | does not fit (44 GB) |
| NVIDIA GeForce RTX 4090 (24 GB) | 1008 GB/s | 252 | 144 | 63 | does not fit (44 GB) |
| AMD Radeon RX 7900 XTX (24 GB) | 960 GB/s | 240 | 137 | 60 | does not fit (44 GB) |
| NVIDIA GeForce RTX 5090 (32 GB) | 1792 GB/s | 448 | 256 | 112 | does not fit (44 GB) |
| NVIDIA DGX Spark (128 GB unified) | 273 GB/s | 68 | 39 | 17 | 8 |
| Apple Mac Studio (M4 Max, 64 GB) | 546 GB/s | 137 | 78 | 34 | 16 |
| Apple Mac Studio (M5 Max, 36 GB, 32-core GPU) | 460 GB/s | 115 | 66 | 29 | does not fit (44 GB; about 32 GB usable) |
| Apple Mac Studio (M5 Max, 64 GB, 40-core GPU) | 614 GB/s | 154 | 88 | 38 | 18 |
| Apple Mac Studio (M5 Ultra, 96 GB) | 1200 GB/s (listed as 1.2 TB/s) | 300 | 171 | 75 | 34 |
Fit rule: 8B Q4 needs 6.8 GB, 14B needs 10.4 GB, 32B needs 21.2 GB, 70B needs 44 GB (params x 0.5 x 1.2 + 2). Mac usable memory subtracts 4 GB. Units in the tok/s columns are a rounded ceiling, not a measured rate.
Why a mixture-of-experts model can generate faster than its total size suggests
A mixture-of-experts (MoE) model stores many expert layers, but only a subset is used for each token. The Qwen3-30B-A3B model card lists 30.5B parameters in total and 3.3B activated, with 128 experts and 8 activated per token (Hugging Face model card README, checked 14 November 2026; the card's config.json matches 128 experts and 8 experts per token). The bytes moved for each generated token follow the active experts plus the shared (non-expert) weights, not the 30.5B total. That is why an MoE model can generate faster than a dense model with the same total parameter count, if the active path is what is read.
Fit is still a total-parameter question on this site: a full GPU load of Qwen3-30B-A3B at Q4_K_M is sized on 30.5B total, which is 20.3 GB by the VRAM formula, not on 3.3B (that would be 3.98 GB and would be wrong). The table above does not give MoE ceilings, because the simple dense formula would understate the bytes moved if you used only active parameters, and overstate them if you used the total. Use the Can I run this? tool for fit; it hides the dense speed line for MoE models.
Related tools and guides
- Can I run this? shows the same Q4_K_M dense bandwidth ceiling when the model fits.
- VRAM calculator applies the params x bytes x 1.2 + 2 formula, including a context control.
- KV cache calculator sizes the extra memory that grows with context length, which the tok/s ceiling above does not include.
- How much VRAM do I need? walks the fit formula by model size.
- KV cache quantization covers OLLAMA_KV_CACHE_TYPE and the llama.cpp cache-type flags.
FAQ
What is the tokens-per-second ceiling on this page?
It is memory bandwidth divided by the bytes of weights read per token, using 0.5 bytes per parameter at Q4_K_M. For an 8B dense model that is 4.0 GB of weights, so a card at 1008 GB/s has a rounded ceiling of 252 tok/s. It is an upper bound from vendor bandwidth and the site constant, not a measured rate.
Why is real speed lower than the ceiling?
The Roofline bound is an upper bound. Memory systems are not 100% efficient, decode still has compute and kernel overhead, and the KV cache plus other working data also use bandwidth. This page does not turn those effects into a second number, because we do not measure them.
Why can a mixture-of-experts model generate faster than its total size suggests?
Only a subset of experts is used for each token. The Qwen3-30B-A3B model card lists 30.5B total parameters and 3.3B activated (128 experts, 8 active per token). The bytes read per token follow the active path, not the full stored total. Fit on this site is still sized on the total.
Why do some 70B cells say "does not fit" even when a ceiling could be computed?
A 70B model at Q4_K_M is 44 GB by the site formula (70 x 0.5 x 1.2 + 2). That is larger than 12, 16, 24, or 32 GB of VRAM, and larger than 36 GB unified memory after a 4 GB macOS reserve. The simple ceiling assumes the weights are read from the memory whose bandwidth is listed. Offload to system RAM is a different bound and is not estimated here.
Does this ceiling include a long context window?
No. The bytes in the denominator are the weight term only (params x 0.5 at Q4_K_M). A long context adds KV cache traffic and memory on top of that. Use the KV cache calculator for the extra memory, and treat a long window as a reason real speed will sit further below the weight-only ceiling.
How does this relate to the VRAM calculator and Can I run this?
The VRAM calculator answers whether the model fits (params x bytes x 1.2 + 2, plus context). This page answers how fast generation can be once the weights fit in the memory whose bandwidth is listed. /can-i-run uses the same Q4_K_M dense ceiling when the model fits, and hides it for MoE or no-fit cases.