The three regimes

There is one idea underneath everything in this course, and it is worth more than any specific technique: a piece of code is slow for exactly one of three reasons, and the fix for each is useless against the other two.

The three regimes

Compute-bound. The hardware is doing arithmetic as fast as it physically can, and the data it needs is already close by. To go faster you need more arithmetic throughput, cheaper arithmetic (lower precision), or less arithmetic.

Memory-bound. The arithmetic units are sitting idle waiting for data to arrive from memory. The number of operations is irrelevant here — what matters is bytes moved. To go faster you move fewer bytes: reuse what you loaded, fuse operations so intermediates never leave the chip, or store the data in a smaller format.

Overhead-bound. Neither the arithmetic nor the memory system is the problem, because the actual work is tiny. The time goes to launching kernels, synchronising, allocating, and dispatching through Python. To go faster you do fewer, bigger operations — batch, fuse, capture the graph.

The trap is that all three feel identical from the outside. Something is slow. The instinct is to make the computation cleverer, which helps in exactly one of the three cases.

Arithmetic intensity is the tell

The ratio that separates the first two regimes is arithmetic intensity: floating point operations performed per byte moved.

intensity  =  FLOPs performed  /  bytes moved

Every processor has a ridge point — the intensity at which its peak arithmetic throughput and its peak memory bandwidth are balanced. Below the ridge, you are memory-bound and only traffic reduction helps. Above it, you are compute-bound.

You compute the ridge point from two numbers on a spec sheet:

ridge point  =  peak FLOP/s  /  peak bytes/s

For a typical modern accelerator this lands somewhere in the range of tens to low hundreds of FLOPs per byte. That number is worth internalising for whatever card you use, because it is the line that decides which half of your instincts apply.

Why this is the right anchor

Consider two operations on the same pair of large matrices.

	Bytes moved	FLOPs	Intensity
Elementwise add	3 × n² × 2	n²	~0.17
Matrix multiply	3 × n² × 2	2n³	~n/3

Identical traffic. The matmul does thousands of times more arithmetic and is nowhere near thousands of times slower, because the add was never limited by arithmetic in the first place — it was limited by the bus, and it saturated the bus.

This is not a toy example. It is the reason attention was rewritten as a memory-movement problem rather than a maths problem, and it is the reason generating a single token from a large language model is slow in a way that buying a faster arithmetic unit does not fix.

The habit this course is built around

Predict before you measure. Every time.

Before every benchmark, write down the number you expect and one sentence saying why. Then measure. The gap between your prediction and reality is the actual lesson, and it is completely invisible if you skip the prediction step — you will simply read the measured number, nod, and learn nothing.

This feels like ceremony for about a week. Then the gaps start closing, and the closing gap is direct evidence that the model in your head now matches the machine. There is no other reliable signal that you have understood something at this level.

Keep a running log of predicted versus actual. It is the most valuable artifact you will produce in the early modules, more valuable than any individual benchmark.

Worked numbers

A100 80GB SXM, the card most papers still quote:

peak FP16 tensor  ≈  312 TFLOP/s
peak HBM2e        ≈  2.0 TB/s
ridge point       ≈  312e12 / 2.0e12  =  156 FLOP/byte

H100 SXM:

peak FP16 tensor  ≈  989 TFLOP/s
peak HBM3         ≈  3.35 TB/s
ridge point       ≈  295 FLOP/byte

L4, a common serving card, is the opposite shape — lots of tensor FLOPs, a thin bus:

peak FP16 tensor  ≈  242 TFLOP/s
peak GDDR6        ≈  300 GB/s
ridge point       ≈  807 FLOP/byte

Write those three ridge points down. Elementwise add (intensity ~0.17) is memory-bound on all of them, and by a wider margin on L4 than on A100. A 4096×4096 matmul (intensity ~1366) is compute-bound on A100 and H100, and still compute-bound on L4. Decode of a 7B model (intensity ~1) is memory-bound on every card you will actually rent this year.

The ridge is climbing. Blackwell-class parts push tensor FLOPs faster than they push HBM, so more of the workloads you used to call "borderline" fall into the memory regime. That is why FlashAttention, KV-cache paging, and weight quantization keep mattering even as advertised TFLOP/s go up.

Launch overhead, in microseconds

A CUDA launch is not free. On a loaded server it is often 5–15 µs of host-side work plus whatever the GPU spends waking a grid. An elementwise kernel on a 4 KB tensor might itself take 3 µs of useful work. The launch is then most of the bill.

That is the third regime with a number on it. If you measure 12 µs for a kernel whose arithmetic and traffic both predict 2 µs, you are not looking at a slow kernel. You are looking at a kernel that should not have been a kernel — fuse it, batch it, or capture it in a CUDA graph.

A useful test: if shrinking the tensor by 4× barely changes wall time, you are overhead-bound. Memory-bound and compute-bound kernels both scale with problem size; launch-bound kernels do not.

Mistakes that look like understanding

"It's compute-bound because it uses a lot of FLOPs." Volume is not intensity. A huge elementwise pass does a huge number of FLOPs and is still memory-bound. Intensity is a ratio.

"We should write a fused kernel" as the first move. Fusion helps the memory regime and the overhead regime. It does nothing for a large matmul that is already sitting on tensor cores. Profile first.

Quoting TFLOP/s for a bandwidth-bound kernel. The number will look terrible and tell you nothing. Report GB/s against the spec sheet. The comparison is only honest in the unit of the scarce resource.

Treating the spec sheet as achievable. Peak tensor FLOP/s assumes dense MMA, right shapes, and no tail effects. Peak HBM assumes a copy kernel written by the vendor. Your kernel will get a fraction. 60–75% of peak bandwidth on a well-written streaming kernel is a good day; 30% of peak tensor FLOPs on a "real" matmul with awkward shapes is common.

Optimising a 200 µs kernel inside a 40 ms decode step. Even a perfect kernel takes the step to 39.8 ms. Find the 12 ms host gap on the timeline first.

In production

The three-regime test is the first sentence of every incident:

"Is the GPU full of math, full of memory transactions, or waiting?"

A serving regression that shows up as "tokens/s dropped 30%" has those three explanations and they want different diffs. Compute: a shape changed and tensor cores stopped firing. Memory: context length doubled and KV traffic ate the bus. Overhead: batch size collapsed and the decode loop became a launch storm.

If you cannot say which regime you are in, you are not debugging yet. You are scrolling a trace.

The rest of this course is that sentence, made mechanical. Module 1 measures it. Module 2 reads it off a profile. Module 3 explains why the hardware has those three ceilings. Modules 4–6 are the levers. Module 7 is what the sentence costs when it is wrong in production.

Checkpoint

Before moving on, be able to answer without notes:

1. Name the three regimes, and one fix that helps in each.
2. What is arithmetic intensity, and what does the ridge point tell you?
3. Two kernels move exactly the same number of bytes, but one does a thousand times more arithmetic. Why might they take a similar amount of time?

The next lesson applies this to language model inference specifically, where the same model explains a serving behaviour that otherwise looks bizarre.

Answers

Try the checkpoint first.

1. Compute-bound — less or cheaper arithmetic (tensor cores, lower precision when the math still needs the FLOPs). Memory-bound — fewer bytes (fuse, quantize, reuse, don't materialise intermediates). Overhead-bound — fewer, bigger launches (batch, fuse, compile, CUDA graph).
2. Intensity is FLOPs / bytes moved. The ridge is peak FLOP/s / peak bytes/s. Below the ridge, only traffic reduction helps; above it, only arithmetic reduction (or faster math units) helps.
3. Both are waiting on the bus. Extra arithmetic is free until you cross the ridge. Identical traffic, similar time.

Go deeper

• Williams, Waterman, Patterson, Roofline: An Insightful Visual Performance Model (2009) — the original ridge-point argument.
• NVIDIA, H100 Tensor Core GPU Architecture — where the 989 TFLOP/s and 3.35 TB/s come from.
• Practice: Roofline Classifier.