Benchmarks
Every number, including the bad one.
Each table names the run it came from. Sizes differ slightly between runs because the builds differ, so figures are not mixed across tables.
| Build | Size | Perplexity |
|---|---|---|
| Q4_K_M | 5.280 GiB | 8.2055 |
| NF4DQ | 4.346 GiB | 8.4181 |
| Q3_K_M | 4.342 GiB | 9.2978 |
Lower is better. NF4DQ is essentially the same size as Q3_K_M while retaining substantially better perplexity; Q4_K_M buys a smaller quality improvement with about 0.93 GiB more weight memory.
| Build | Size | Prefill | Decode |
|---|---|---|---|
| NF4DQ | 13.23 GiB | 734.52 | 60.81 |
| Q4_K_M | 17.66 GiB | 1300.04 | 48.67 |
NF4DQ is 25 percent smaller and generates 25 percent faster. Prefill is roughly half the speed because prompt processing takes a fallback path until the format has a tile loader.
| Build | Size | Perplexity |
|---|---|---|
| BF16 | 50.900 GiB | 6.9530 |
| Q4_K_M | 15.652 GiB | 6.9787 |
| NF4DQ | 13.230 GiB | 7.0890 |
| Q3_K_M | 12.643 GiB | 7.3362 |
Lower is better. NF4DQ sits between Q4_K_M and Q3_K_M on quality while being smaller than both.
| Machine | Model | Decode | With MTP |
|---|---|---|---|
| NVIDIA L4, 23 GB | 27B | 16.84 | 30.13 |
| Apple M1, 8 GB | 9B | 10.28 | – |
MTP is multi-token prediction. The L4 row is worth reading twice: a 27B model in about 16.9 GB total with an 8k context, on a card that Q4_K_M would leave almost nothing spare on.
If you reproduce any of this and get something different, open an issue with your hardware and build flags. Numbers that do not replicate are worth more to this project than numbers that do.