How to Tune llama.cpp on 8GB VRAM: Why 32K Is Safer and 64K Needs KV Cache Quantization

Thu, 23 Apr 2026 12:13:04 +0800

Whether 8GB of VRAM is enough to run local LLMs smoothly, especially under long-context workloads, is one of the most common questions people run into when using llama.cpp.

There are three key takeaways worth remembering first:

On 8GB VRAM, 32K context is usually the safer balance point
If you really want to run 64K, KV Cache quantization is often essential
In full-GPU inference, blindly increasing CPU thread count can actually make performance worse

1. First, what do 32K, 64K, and KV Cache actually mean?

For many readers, these are the three terms that cause the most confusion.

32K and 64K refer to context length, meaning how many tokens the model can process at one time. Here, K means thousand, so 32K is about 32000 tokens, and 64K is about 64000 tokens. The longer the context, the more prior content the model can see at once, which is useful for long-document QA, long conversations, and multi-step analysis.

KV Cache is an intermediate-result cache that the model keeps in order to speed up autoregressive generation. You can think of it like this: once the model has already read and computed part of the context, it does not need to recompute everything from scratch every time. Instead, it stores key intermediate information and reuses it. The K and V come from Key and Value in the Transformer architecture.

Why do these three terms always appear together? Because:

32K and 64K define how much content you want the model to remember at once
KV Cache determines how much extra VRAM is needed to maintain that memory
The longer the context, the larger the KV Cache usually becomes, and the higher the VRAM pressure gets

So when long-context inference slows down, the root problem is often not that the model is “bad at computing”, but that the cache has grown large enough to push VRAM to its limit.

2. Why does 32K perform so differently from 64K?

Using roughly 30000 Chinese characters from The Three-Body Problem as a stress-test input, the comparison between 32K and 64K context can look dramatic: with similar document size, 64K can become much slower and total runtime can increase significantly.

The reason is not that the model suddenly becomes worse. The real issue is hitting the VRAM boundary.

At 32K, model weights plus cache may still fit within 8GB VRAM, so most data traffic stays on the GPU’s own memory bandwidth. But once you move to 64K, the cache grows further, total memory use approaches or exceeds the VRAM ceiling, and part of the data gets pushed into shared or system memory.

At that point, what collapses is not raw compute, but bandwidth.

In other words, what looks like “context doubled and performance crashed” is often really a case of the data path falling out of VRAM and into much slower memory.

3. If you want 64K, KV Cache quantization matters a lot

One of the most important conclusions for 8GB VRAM users is that KV Cache quantization matters a great deal.

Without changing the model itself, quantizing only the cache can directly reduce cache memory usage under long context. That means some of the data that previously spilled out of VRAM can move back into VRAM. As a result, 64K is still heavier than 32K, but it is less likely to fall into the slowest performance zone.

Put simply:

32K is the more practical default range for 8GB VRAM
64K is not impossible
But without cache quantization, performance can drop from “usable” to “hard to use”

If your goal is stable long-context inference, the usual priority should be:

Check whether VRAM is already near its ceiling
Decide whether to enable KV Cache quantization
Only then continue experimenting with more aggressive throughput settings

4. Low GPU utilization does not mean the GPU is idle

This is a point that often breaks intuition.

When people see only 20% or 30% GPU usage in Task Manager, they often assume:

the parameters must be wrong
the model is not really running on the GPU
the GPU is not being used fully

But the more likely explanation in llama.cpp inference is that the bottleneck is not core compute, but memory reads and writes.

That means GPU cores may finish a batch of computation quickly, then spend the rest of the time waiting for the next batch of weights or cached data to arrive.

So what you see becomes:

core utilization is not especially high
but end-to-end speed still fails to improve

This is not the GPU being lazy. It is the data path being too narrow.

That is why you should not look only at GPU Usage when judging local LLM performance. VRAM capacity, memory bandwidth, and cache spillover often matter more.

5. Increasing throughput parameters can help, but only if VRAM can handle it

Another useful idea is this: if GPU cores are not fully saturated, maybe you can increase throughput-related parameters so the GPU processes more data at once and uses its parallelism more effectively.

This can indeed improve speed.

But there is an important condition: VRAM must still have headroom.

Because once you increase throughput-related settings, you often also increase VRAM usage. If you are already in a 64K scenario with large cache and VRAM near exhaustion, pushing those parameters further can lead to two outcomes:

a crash
or a fallback into much slower shared-memory behavior

So the safer sequence is usually not “max out the knobs first”, but:

protect the VRAM boundary first
then try throughput optimization
after every change, check both speed and stability again

6. More CPU threads are not always better

This is one of the easiest traps to remember.

It is very natural to assume that more threads should mean better speed. But in practice, once the model is already running mostly on the GPU, forcing CPU thread count higher can make performance noticeably worse.

The reason is straightforward.

In full-GPU inference, the CPU is more of a scheduler and preprocessing helper than the main compute engine. If you open too many threads, CPU-side thread contention, scheduling overhead, and context-switching costs all become heavier, which can disrupt the data flow that should have stayed smooth.

The result is:

the CPU looks busier
but overall speed gets slower

So in this kind of setup, default settings or lower thread counts are often more reliable than simply maxing everything out.

7. A more practical approach for 8GB VRAM users

If we compress the conclusions above into a practical workflow, it looks roughly like this:

1. Treat 32K as the default goal

If you only have an 8GB GPU, do not rush to chase 64K. 32K is usually the more realistic balance between speed, stability, and memory usage.

2. If you want 64K, deal with the cache first

Do not start by asking whether you can squeeze out a little more speed. First confirm whether KV Cache is quantized and whether VRAM is already near the limit.

3. Do not judge everything by GPU utilization

Low utilization does not necessarily mean the settings are wrong. It may simply mean memory bandwidth is the real bottleneck.

4. Throughput optimization is valid, but do not cross the VRAM boundary

These parameters can help, but only if there is still enough VRAM headroom.

5. Be conservative with CPU threads first

If the model is already running mostly on the GPU, higher CPU thread counts are not automatically better. Start with defaults or lower thread counts, then test gradually.

Conclusion

The most valuable part of this whole discussion is not just a few benchmark numbers, but the fact that it makes one easily overlooked truth much clearer:

Local LLM tuning is often not about pushing every setting to the maximum. It is about understanding whether your real bottleneck is compute, VRAM capacity, memory bandwidth, or CPU scheduling.

For 8GB VRAM users, the safer strategy is usually not to force the longest possible context, but to protect the VRAM boundary first and only then decide how far to push further.

If you only remember one sentence, make it this:

32K is often the more stable working range for 8GB VRAM; 64K is possible, but only if you have already brought KV Cache and VRAM usage under control.

GPU Tuning on KnightLi Blog