https://www.knightli.com/en/tags/llama/ Recent content in Llama on KnightLi Blog Hugo -- gohugo.io en Sat, 11 Apr 2026 20:07:29 +0800 https://www.knightli.com/en/2026/04/11/llama-gguf-quantization-selection/ Sat, 11 Apr 2026 20:07:29 +0800 https://www.knightli.com/en/2026/04/11/llama-gguf-quantization-selection/ <p>When selecting a Llama GGUF model on Hugging Face, you can think of quantization levels like resolution: lower levels need less VRAM/RAM, but quality drops gradually.</p> <h2 id="understand-32-16-and-q-levels-first">Understand 32, 16, and Q levels first </h2><ul> <li><code>32</code>: closest to original/uncompressed quality, but hardware demand is extreme.</li> <li><code>16</code>: still very close to original quality, around half the size of <code>32</code>.</li> <li><code>Q8</code>: common entry point for quantized models (<code>Q8_0</code> or <code>Q8</code>).</li> <li><code>Q6</code>, <code>Q5</code>, <code>Q4</code>, <code>Q3</code>, <code>Q2</code>: lower number means lower resource use and higher quality loss risk.</li> </ul> <h2 id="what-k_m--k_s-means">What <code>K_M</code> / <code>K_S</code> means </h2><p><code>K_M</code> and <code>K_S</code> are mixed quantization variants:</p> <ul> <li>most weights stay at the target quantization level</li> <li>important parts keep higher precision</li> </ul> <p>So at the same level, <code>Qx_K_M</code> or <code>Qx_K_S</code> is usually slightly better than plain <code>Qx</code>.</p> <h2 id="practical-picking-strategy">Practical picking strategy </h2><ul> <li>If hardware allows, start with <code>Q8</code>.</li> <li>If memory is tight, step down through <code>Q6</code> / <code>Q5</code> / <code>Q4</code>.</li> <li>Try not to go below <code>Q4</code>; <code>Q4_K_M</code> is a common lower bound.</li> <li>Below <code>Q4</code>, quality degradation becomes increasingly visible.</li> </ul> <h2 id="quality-order-best-to-worst">Quality order (best to worst) </h2><ul> <li><code>32</code></li> <li><code>16</code></li> </ul> <p>&ndash; Above this point, quality is effectively the same, but hardware requirements are extreme &ndash;</p> <ul> <li><code>Q8</code></li> <li><code>Q6_K_M</code></li> <li><code>Q6_K_S</code></li> <li><code>Q6</code></li> <li><code>Q5_K_M</code></li> <li><code>Q5_K_S</code></li> <li><code>Q5</code></li> </ul> <p>&ndash; This is the typical sweet spot &ndash;</p> <ul> <li><code>Q4_K_M</code></li> <li><code>Q4_K_S</code></li> <li><code>Q4</code></li> </ul> <p>&ndash; Below this point, quality loss becomes visible &ndash;</p> <ul> <li><code>Q3_K_M</code></li> <li><code>Q3_K_S</code></li> <li><code>Q3</code></li> <li><code>Q2_K_M</code></li> <li><code>Q2_K_S</code></li> <li><code>Q2</code></li> </ul> <p>If you want one short rule: start with <code>Q8</code> or <code>Q6_K_M</code>, then move down to <code>Q5</code> or <code>Q4_K_M</code> only when needed.</p>