The three regimes
Every workload is waiting on arithmetic, on bytes, or on the launch queue. Optimising the wrong one buys nothing, and almost everyone optimises the wrong one first.
By the end
Name the three things a kernel can be limited by, and say which one a given workload is in before touching a profiler.
There is one idea underneath everything in this course, and it is worth more than any specific technique: a piece of code is slow for exactly one of three reasons, and the fix for each is useless against the other two.
The three regimes
Compute-bound. The hardware is doing arithmetic as fast as it physically can, and the data it needs is already close by. To go faster you need more arithmetic throughput, cheaper arithmetic (lower precision), or less arithmetic.
Memory-bound. The arithmetic units are sitting idle waiting for data to arrive from memory. The number of operations is irrelevant here — what matters is bytes moved. To go faster you move fewer bytes: reuse what you loaded, fuse operations so intermediates never leave the chip, or store the data in a smaller format.
Overhead-bound. Neither the arithmetic nor the memory system is the problem, because the actual work is tiny. The time goes to launching kernels, synchronising, allocating, and dispatching through Python. To go faster you do fewer, bigger operations — batch, fuse, capture the graph.
The trap is that all three feel identical from the outside. Something is slow. The instinct is to make the computation cleverer, which helps in exactly one of the three cases.
Arithmetic intensity is the tell
The ratio that separates the first two regimes is arithmetic intensity: floating point operations performed per byte moved.
intensity = FLOPs performed / bytes moved
Every processor has a ridge point — the intensity at which its peak arithmetic throughput and its peak memory bandwidth are balanced. Below the ridge, you are memory-bound and only traffic reduction helps. Above it, you are compute-bound.
You compute the ridge point from two numbers on a spec sheet:
ridge point = peak FLOP/s / peak bytes/s
For a typical modern accelerator this lands somewhere in the range of tens to low hundreds of FLOPs per byte. That number is worth internalising for whatever card you use, because it is the line that decides which half of your instincts apply.
Why this is the right anchor
Consider two operations on the same pair of large matrices.
| Bytes moved | FLOPs | Intensity | |
|---|---|---|---|
| Elementwise add | 3 × n² × 2 | n² | ~0.17 |
| Matrix multiply | 3 × n² × 2 | 2n³ | ~n/3 |
Identical traffic. The matmul does thousands of times more arithmetic and is nowhere near thousands of times slower, because the add was never limited by arithmetic in the first place — it was limited by the bus, and it saturated the bus.
This is not a toy example. It is the reason attention was rewritten as a memory-movement problem rather than a maths problem, and it is the reason generating a single token from a large language model is slow in a way that buying a faster arithmetic unit does not fix.
The habit this course is built around
Predict before you measure. Every time.
Before every benchmark, write down the number you expect and one sentence saying why. Then measure. The gap between your prediction and reality is the actual lesson, and it is completely invisible if you skip the prediction step — you will simply read the measured number, nod, and learn nothing.
This feels like ceremony for about a week. Then the gaps start closing, and the closing gap is direct evidence that the model in your head now matches the machine. There is no other reliable signal that you have understood something at this level.
Keep a running log of predicted versus actual. It is the most valuable artifact you will produce in the early modules, more valuable than any individual benchmark.
Checkpoint
Before moving on, be able to answer without notes:
- Name the three regimes, and one fix that helps in each.
- What is arithmetic intensity, and what does the ridge point tell you?
- Two kernels move exactly the same number of bytes, but one does a thousand times more arithmetic. Why might they take a similar amount of time?
The next lesson applies this to language model inference specifically, where the same model explains a serving behaviour that otherwise looks bizarre.