Inference Engineering
Module 00·Orientation·5 min read

Why inference is asymmetric

Reading a prompt and writing a reply are the same model doing the same maths, and they sit in opposite performance regimes. Almost every serving decision follows from that one fact.

By the end

Explain prefill versus decode, why decode is bandwidth-bound, what the KV cache actually caps, and why bigger batches trade latency for throughput.

Generating text happens in two phases with completely different performance characteristics. Confusing them is the single most common mistake when reasoning about inference, and separating them explains almost every serving decision you will meet.

Prefill and decode

Prefill processes the prompt. Every token in the prompt can be handled at once, so the work is large, parallel, and arithmetic-heavy. Prefill is compute-bound — the classic regime, where more arithmetic throughput genuinely makes it faster.

Decode generates the output, one token at a time. Each token depends on the one before it, so there is no parallelism across the sequence. To produce a single token, the hardware reads every weight in the model and does a comparatively tiny amount of arithmetic with each one.

That is an arithmetic intensity near the floor. Decode is memory-bandwidth-bound.

The consequence is stark. For a model with W bytes of weights on a card with B bytes per second of memory bandwidth, the floor on time per output token is:

minimum time per output token  =  W bytes of weights  /  B bytes per second

Nothing about your code gets under that line at batch size one. A 14 GB model on a card with 600 GB/s of bandwidth cannot emit tokens faster than about 23 ms apart, no matter how good the kernels are, because that is how long it takes to read the weights once.

This is worth sitting with. The dominant cost of generating a token is not the arithmetic. It is reading the model.

What follows immediately

Once you accept that decode is bandwidth-bound, a set of otherwise-strange behaviours become obvious:

Quantization helps decode far more than prefill. Halving the bytes per weight halves the traffic, and traffic is the binding constraint. In prefill, where you were compute-bound, the same change helps much less.

Batching is close to free during decode — up to a point. You already paid to read the weights. Running a second sequence through them costs almost no extra traffic, so throughput scales with batch size while the per-token latency barely moves. This is why serving systems work so hard to keep batches full.

Speculative decoding is a regime conversion. A small draft model proposes several tokens, and the large model verifies all of them in one pass. You have turned a sequence of bandwidth-bound steps into one compute-bound step, which is a much better place to be.

A faster card with the same bandwidth does nothing for decode. Buying arithmetic to fix a memory problem is the exact error the three-regimes model exists to prevent.

The KV cache, and what it actually limits

To avoid recomputing attention over the whole sequence at every step, the keys and values for previous tokens are cached. That cache grows with sequence length and with the number of concurrent sequences.

The important part: the KV cache, not the weights, is usually what caps how many requests fit on a device. The weights are a fixed cost paid once. The cache is a per-request, per-token cost that grows as conversations get longer.

This reframes a capacity question people usually get wrong. "How many users fit on this GPU" is rarely answered by model size. It is answered by how much memory is left after the weights, divided by how much cache each concurrent sequence consumes — which depends on how long you let sequences get.

It also explains why memory fragmentation became a headline problem, and why managing the cache in fixed pages, the way an operating system manages virtual memory, was a significant advance rather than an implementation detail.

The metrics, and the tension between them

Three numbers describe a serving system:

  • TTFT — time to first token. Dominated by prefill.
  • TPOT or ITL — time per output token. Dominated by decode.
  • Throughput — total tokens per second across all concurrent requests.

And one tension governs every configuration decision:

Bigger batches raise throughput and hurt latency.

Every serving configuration is a position on that curve. There is no setting that is simply best; there is only a choice about which end of the curve your product needs. A chat interface cares about TTFT and TPOT because a human is waiting. A bulk document pipeline cares only about throughput and will happily accept far worse per-request latency to get it.

If you take one thing into an interview or a design review from this module, take this: "it's slow" is not a diagnosis. Slow at what — first token, or each subsequent token? Under load, or alone? Those are different problems with different fixes.

Checkpoint

Explain out loud, without notes, in four sentences:

  1. What prefill and decode each do, and which regime each sits in.
  2. Why decode is bandwidth-bound.
  3. What the KV cache limits, and why it — not the weights — usually caps concurrency.
  4. Why bigger batches buy throughput and cost latency.

If you can do that cleanly, you have the vocabulary to hold a real conversation about inference performance. The rest of the course is about making it concrete enough to act on — starting, in the next module, with measuring these effects yourself in PyTorch.