Arithmetic intensity, measured
Two kernels move identical bytes; one does thousands of times more arithmetic and is nowhere near thousands of times slower. Measure why, then derive your own card's ridge point.
By the end
Compute a kernel's arithmetic intensity, decide which regime it is in from your own measurements, and locate your card's ridge point.
Run it
This lesson has a notebook. Read the page first, then work through the cells — every # PREDICT: marker is a place to commit a number before you run it.
In Colab: File ▸ Upload notebook, then set the runtime to GPU. Locally: jupyter lab.
Module 0 introduced arithmetic intensity as an argument. This lesson turns it into numbers you produced yourself, on your own hardware.
The experiment
Take two large float16 matrices and run two operations on them:
a + b # elementwise add
a @ b # matrix multiply
Both touch the same three tensors — read a, read b, write a result — so the memory traffic is identical. The FLOP counts are not remotely identical:
| Bytes moved | FLOPs | Intensity | |
|---|---|---|---|
a + b | 3n²·2 | n² | ~0.17 FLOP/byte |
a @ b | 3n²·2 | 2n³ | ~n/3 FLOP/byte |
At n=4096 the matmul does roughly 8000× more arithmetic. Predict the slowdown before you measure it. Almost nobody predicts this one correctly the first time.
Reading your own numbers
The notebook converts both measurements into the rate that suits each kernel:
- The add is reported as GB/s, and lands close to what the card can physically sustain.
- The matmul is reported as TFLOP/s, and lands at a meaningful fraction of peak.
That pairing is the whole point. Each kernel is running near a ceiling — just a different ceiling. The add is not badly written; it is finished, in the sense that no amount of cleverness will make memory faster. The only remaining lever on it is moving fewer bytes.
When a kernel's achieved bandwidth approaches the card's peak bandwidth, stop optimising the computation. You are reading the machine at its limit. Either reduce traffic or accept the number.
The ridge point
Two numbers from your card's spec sheet give you the line between the regimes:
ridge point = peak FLOP/s / peak bytes/s
Compute it for the card you are using and write it down. Any kernel whose intensity falls below that value is memory-bound, and no rewrite of its arithmetic will help. Any kernel above it is compute-bound, and traffic reduction will not help.
Then sweep matmul across sizes and watch intensity climb past the ridge. Small matmuls sit below it — which is a second, independent reason small matrices perform badly, alongside the launch overhead from the previous lesson.
Why this explains token generation
Now return to the number from lesson one: a 7B model in bfloat16 is about 14 GB of weights.
Generating one token reads all of them, and does roughly two FLOPs per parameter. That is an intensity of about 1 FLOP per byte — far below any modern card's ridge point, which lives in the tens to low hundreds.
So the floor on time per output token is set by bandwidth alone:
minimum time per output token = 14 GB / card bandwidth
On a card with 600 GB/s that is roughly 23 ms, and no kernel engineering gets underneath it at batch size one. This is not a claim you have to take on faith any more — it is the same calculation you just ran on a + b, applied to a bigger tensor.
It also tells you precisely which levers exist: make the weights smaller (quantize), read them once for several sequences (batch), or read them fewer times per token accepted (speculative decoding). All three attack traffic, because traffic is the constraint. Everything else is noise.
Checkpoint
- Two kernels move the same bytes; one does 8000× more arithmetic and is 40× slower. Explain.
- What is your card's ridge point, and what does a kernel below it tell you to do?
- Estimate the minimum time per output token for a 70B model in
int8on a card with 2 TB/s of bandwidth.
Practice this lesson
The reading is the model. These drills are the hours — 2 problems that force the numbers onto paper before the next lesson.