Comparing llama.cpp GPU Performance: CUDA, ROCm, Vulkan, and Reproducible Benchmarks

The llama.cpp runs in the community are well suited to find candidates for a graphic card, but they cannot be directly considered as a purchase list. A different driver, backend, submission version, utility, model quantification, context and visible offload ratio may change the outcome.

This document retains the original scoreboard reference entry, but does not copy hundreds of runs. The focus should be changed to: how to read the indicators, how to repeat them, and under what circumstances the two sets of figures could be compared.

Start with pp512 and tg128

llama-bench Two common:

• pp512: one-time processing of 512 tips token, close to prompt ingingration/prefill.
• tg128: A continuous generation of 128 tokens, close to decode when the user is waiting to answer.

The unit is usually tokens/s, but the two metrics expose different bottlenecks. Batch processing and long prompt ingestion depend more on pp; chat and code completion depend more on tg. A single larger number is not enough to claim that one GPU is “twice as fast.”

Q4_0 and Q4_K_M are different quantizations. Model size, VRAM bandwidth pressure, and compute paths also vary. FA indicates whether Flash Attention is enabled, which can materially affect long-context performance on a given backend.

Which runs are directly comparable?

At least meet at the same time:

If scoreboard lacks several of these, it can only be used for rough screening and cannot be accurately calculated.

Complete CUDA Leaderboard

Llama 2 7B, Q4_0, no FA

Llama 2 7B, Q4_0, with FA

Establish your own test baseline

Record version:

1
2

llama-bench --version
git -C llama.cpp rev-parse HEAD

Record NVIDIA environment:

1

nvidia-smi --query-gpu=name,driver_version,memory.total,power.limit,temperature.gpu --format=csv

AMD Linux:

1
2

rocminfo | head -80
rocm-smi --showproductname --showmeminfo vram --showpower --showtemp

Vulkan:

1

vulkaninfo --summary

These outputs should be saved with the rundown CSV. Only t/s has no environmental records, and it will be difficult to recover in a few weeks.

Minimal repeat command

Assuming the same test. model-q4_k_m.gguf:

1
2
3
4
5
6
7
8

llama-bench \
  -m ./models/model-q4_k_m.gguf \
  -p 512 \
  -n 128 \
  -ngl 999 \
  -fa 1 \
  -r 5 \
  -o csv > bench-fa.csv

Close Flash Attention

1
2
3
4
5
6
7
8

llama-bench \
  -m ./models/model-q4_k_m.gguf \
  -p 512 \
  -n 128 \
  -ngl 999 \
  -fa 0 \
  -r 5 \
  -o csv > bench-no-fa.csv

Run one preheat and repeat at least five times. Do not download models at the same time, run games or allow desktop applications to occupy large amounts of displays.

Verify that the model is fully offloaded to the GPU

Observation at test:

1

nvidia-smi dmon -s pucm

If the model spills over the system or only partially unmounts, the result reflects a combination of CPU, PCIE and GPU and is no longer a simple graphic card function. Make sure before you compare the two cards. -ngl And the actual offload status is the same.

Multi GPU also needs to record spit Mode, card display, PCIe/SXM interconnectivity and scalding. Two cards can fit a larger model, which does not mean that a small model decode must be faster than a single large visible card.

Choosing CUDA, ROCm, or Vulkan

CUDA

NVIDIA usually has the most mature llama.cpp path. In addition to t/s, the cards are selected for visual storage capacity, visible bandwidth, utility and second-hand card reliability. Target models are often more important than peak computing.

ROCM/HIP

AMD data must have ROCm and driver versions. The consumer-level Radeon support matrix, system version and specific architecture will affect the proper functioning. Do not extend the achievement of a card in a Linux combination directly to Windows.

Vulkan

Vulkan has a wide range of coverage, suitable for equipment lacking original CUDA/ROCm paths, and for cross-platform baselines. However, the optimization of different drivers and equipment varies widely, and community tables in particular need to be revisited.

Use the results to answer the actual GPU-selection question

Determines the target model and context, then estimates the weight, KV Cache and running time balance. Visibility is simply equal to the size of the model file.

It is recommended that the selection be judged in order:

1. Whether or not the target quantification can be fully captured.
2. Whether there is room for KV Cache in the required context.
3. tg Whether interactive speed is satisfied.
4. pp Whether or not to meet the long hint of vomiting.
5. The back end is stable on your operating system.
6. Admissibility of utility, noise and total cost.

For example, one. tg128 Higher, but less visible, cards that run the target 30B model may not be as fast as a card that is fully loaded.

What a useful result sheet should include

It is suggested that at least:

1
2

tested_at,llama_commit,gpu,driver,backend,model,quant,ngl,flash_attention,pp512_tps,tg128_tps,power_limit,notes
2026-08-02,abcdef1,Example GPU,driver-version,CUDA,model-name,Q4_K_M,999,true,0,0,250W,replace zeros with measured median

When the results are made public, the figures are measured, medium or quoted. Do not mix the records of multiple authors, multiple years, multiple models and continue to be referred to as the “complete ladder”.

How to use community benchmark data

The scoreboard in GitHub Discussions still has value: it helps to discover whether a card is passing, whether the back end is mature, and broadly performance ranges. However, the same model should be used to rescale candidate hardware before eventual purchase or deployment.

Original reference:

• CUDA scoreboard discussion #15013
• Apple Silicon discussion #4167
• ROCm scoreboard discussion #15021
• Vulkan scoreboard discussion #10879
• llama.cpp GitHub