I Thought My Benchmark Found an Effect, but It Was Measurement Noise

Introduction

In part 2, I decided not to use KV cache quantization. Here’s the reason I gave:

q8_0 shows perplexity on par with f16, with almost no quality loss. But where f16 fits, f16 is faster, because dequantization runs on every token.

That decision had a weak spot: its basis was someone else’s measurement. I had read a report that “at 64K depth it drops to 55% of f16”, and rejected it without measuring on my own machine.

Since then I had widened the context from 16K to 64K, so I needed to check that assumption. I measured it. And at one point, I reached the wrong conclusion. This is the record of that.

What to Measure

llama-bench has -d (n-depth). It starts generating with the given number of tokens already in the KV cache, so you can directly measure “how much slower generation gets as the conversation grows”.

llama-bench -hf <model> -ngl 99 -fa 1 -b 8192
            -p 0 -n 128 -d 0,4096,16384,32768

-p 0 removes prompt processing and isolates generation alone.

I measured with f16 first, then did the same with -ctk q8_0 -ctv q8_0.

Round 1: The Decision Was Overturned (Or So I Thought)

Depth	f16	q8_0	Difference
0	16.79	18.36	+9.4%
4,096	16.27	18.17	+11.7%
16,384	15.05	16.54	+9.9%
32,768	14.29	15.38	+7.6%

q8_0 was faster at every depth. And with less variance, too.

At that point, I had even come up with an explanation.

This machine has been consistently bandwidth-bound. MoE being 5.7x faster than dense 27B, -b helping while -ub doesn’t: bandwidth explained all of it. Quantizing KV reduces reads. On a bandwidth-bound machine, maybe q8_0 comes out ahead.

It sounds plausible. The report part 2 relied on was probably from a dGPU, which is compute-bound, so the dequantization cost is a straight loss there. Here it’s bandwidth-bound, so it comes out ahead…

What follows is about how the more plausible the explanation, the more dangerous it is.

What Bothered Me

Looking over the table, the 9.4% difference at depth 0 bothered me.

At depth 0, the KV cache is empty. Quantizing an empty cache shouldn’t change the speed, since there’s nothing to read.

If the bandwidth-bound hypothesis were right, the gap should grow at greater depths and be close to zero at depth 0. The actual table was the opposite: the gap was largest at depth 0 and shrank as it got deeper.

The explanation and the observations didn’t match.

Round 2: Measuring Again

llama-bench accepts comma-separated conditions. Within a single launch, I measured f16 and q8_0 alternately, 5 times each.

llama-bench ... -d 0,32768 -ctk f16,q8_0 -ctv f16,q8_0 -r 5

The result flipped.

type_k	type_v	Depth 0	Depth 32,768
f16	f16	18.57 ± 0.51	15.65 ± 0.23
f16	q8_0	17.60 ± 0.99	14.14 ± 0.41
q8_0	f16	17.58 ± 0.35	15.10 ± 0.35
q8_0	q8_0	17.71 ± 1.03	14.06 ± 0.37

f16 is the fastest. At 32K depth it’s +11.3% over q8_0 (15.65 vs. 14.06), and the error ranges don’t overlap.

And at depth 0, the difference disappeared. The ranges of 18.57 ± 0.51 and 17.71 ± 1.03 overlap. Just as physics says: if the KV cache is empty, quantization has no effect.

The decision in part 2 was right after all.

What Was Going On

Between round 1 and round 2, the value for the same f16 / depth 0 setting differs.

1 本目: 16.79
3 本目: 18.57   ← 同じ設定なのに +10.6%

There was about 10% variation between runs. The difference I was trying to measure (about 9%) was buried entirely inside it.

Runs 1 and 2 were separate processes, run one after the other. In between, the warm-up state, power, and thermals all change. Run 2 just happened to run under more favorable conditions.

The observation that “q8_0 is faster” was really showing a difference in order, not in settings.

The Bandwidth-Bound Hypothesis Was Wrong, Too

On top of that, the explanation I came up with was wrong as well.

Quantizing KV does reduce reads. But the dequantization cost outweighed it. Even on a bandwidth-bound machine.

A smaller finding: quantizing V costs more than quantizing K.

	Depth 32,768
Both K and V f16	15.65
K only q8_0	15.10
V only q8_0	14.14
Both K and V q8_0	14.06

K alone loses 3.5%; V alone loses 9.7%. If you use q8_0, start with K: that seems worth remembering.

What I Gained: Slowdown with Depth Was Gradual

Mistakes aside, I did learn something new.

Depth	tg128	vs. depth 0
0	16.79	—
4,096	16.27	−3%
16,384	15.05	−10%
32,768	14.29	−15%

Even with 32K filled, it’s only 15% lower. Compared with the report part 2 relied on, “55% of f16 at 64K depth”, the slope is much gentler.

So even when using deep context for long conversations or code reviews, speed is not much of a concern.

These 4 points are values from a single launch where only the depth varied, so they can be compared with each other. I’m not using the method I just criticized here.

Lessons

When the difference is around 10%, check the measurement noise first.

Before measuring the size of an effect, I should have looked at how much the same setting moves when measured twice. If I had done that, I’d have known the moment I saw the first table that “this difference is within the noise”.

Benchmarks from separate launches can’t be compared.

Not just llama-bench: the moment you split processes, the warm-up, power, thermal, and cache states no longer line up. Vary the conditions you want to compare within a single launch.

-ctk f16,q8_0 -ctv f16,q8_0 -r 5

Be most suspicious when an explanation comes to mind.

The most dangerous part this time was that the explanation “it’s bandwidth-bound, so q8_0 wins” came to me right away. And it was consistent with my earlier observations.

A plausible explanation lends credibility to wrong numbers. What saved me was that the explanation carried another prediction (there should be no difference at depth 0). That prediction failed, which is what made me measure again.

When you think you’ve explained something, check what else that explanation predicts. If that doesn’t match, either the explanation or the numbers are wrong.

Summary

• When I re-checked a past decision (keep KV at f16), I reached the opposite conclusion at first
• I noticed that a difference at depth 0 was physically odd, measured again, and the original decision was right
• The cause was 10% variation between runs. The difference I was measuring was buried in it
• No settings changed. What I gained was that slowdown at 32K depth is a gentle 15%
• If you use q8_0, start with K. Quantizing V costs more

It was a story that ended with “it was right after all”, but for me, the process of getting it wrong along the way was the bigger takeaway.