Why inference is asymmetric

Generating text happens in two phases with completely different performance characteristics. Confusing them is the single most common mistake when reasoning about inference, and separating them explains almost every serving decision you will meet.

Prefill and decode

Prefill processes the prompt. Every token in the prompt can be handled at once, so the work is large, parallel, and arithmetic-heavy. Prefill is compute-bound — the classic regime, where more arithmetic throughput genuinely makes it faster.

Decode generates the output, one token at a time. Each token depends on the one before it, so there is no parallelism across the sequence. To produce a single token, the hardware reads every weight in the model and does a comparatively tiny amount of arithmetic with each one.

That is an arithmetic intensity near the floor. Decode is memory-bandwidth-bound.

The consequence is stark. For a model with W bytes of weights on a card with B bytes per second of memory bandwidth, the floor on time per output token is:

minimum time per output token  =  W bytes of weights  /  B bytes per second

Nothing about your code gets under that line at batch size one. A 14 GB model on a card with 600 GB/s of bandwidth cannot emit tokens faster than about 23 ms apart, no matter how good the kernels are, because that is how long it takes to read the weights once.

This is worth sitting with. The dominant cost of generating a token is not the arithmetic. It is reading the model.

What follows immediately

Once you accept that decode is bandwidth-bound, a set of otherwise-strange behaviours become obvious:

Quantization helps decode far more than prefill. Halving the bytes per weight halves the traffic, and traffic is the binding constraint. In prefill, where you were compute-bound, the same change helps much less.

Batching is close to free during decode — up to a point. You already paid to read the weights. Running a second sequence through them costs almost no extra traffic, so throughput scales with batch size while the per-token latency barely moves. This is why serving systems work so hard to keep batches full.

Speculative decoding is a regime conversion. A small draft model proposes several tokens, and the large model verifies all of them in one pass. You have turned a sequence of bandwidth-bound steps into one compute-bound step, which is a much better place to be.

A faster card with the same bandwidth does nothing for decode. Buying arithmetic to fix a memory problem is the exact error the three-regimes model exists to prevent.

The KV cache, and what it actually limits

To avoid recomputing attention over the whole sequence at every step, the keys and values for previous tokens are cached. That cache grows with sequence length and with the number of concurrent sequences.

The important part: the KV cache, not the weights, is usually what caps how many requests fit on a device. The weights are a fixed cost paid once. The cache is a per-request, per-token cost that grows as conversations get longer.

This reframes a capacity question people usually get wrong. "How many users fit on this GPU" is rarely answered by model size. It is answered by how much memory is left after the weights, divided by how much cache each concurrent sequence consumes — which depends on how long you let sequences get.

It also explains why memory fragmentation became a headline problem, and why managing the cache in fixed pages, the way an operating system manages virtual memory, was a significant advance rather than an implementation detail.

The metrics, and the tension between them

Three numbers describe a serving system:

• TTFT — time to first token. Dominated by prefill.
• TPOT or ITL — time per output token. Dominated by decode.
• Throughput — total tokens per second across all concurrent requests.

And one tension governs every configuration decision:

Bigger batches raise throughput and hurt latency.

Every serving configuration is a position on that curve. There is no setting that is simply best; there is only a choice about which end of the curve your product needs. A chat interface cares about TTFT and TPOT because a human is waiting. A bulk document pipeline cares only about throughput and will happily accept far worse per-request latency to get it.

If you take one thing into an interview or a design review from this module, take this: "it's slow" is not a diagnosis. Slow at what — first token, or each subsequent token? Under load, or alone? Those are different problems with different fixes.

Worked decode floors

Take the bandwidth-bound floor seriously by computing it for hardware you will actually use.

A 7B dense model in bfloat16 is ~14 GB of weights. Ignoring KV for a moment:

Card	Bandwidth	Floor, batch=1	Tokens/s ceiling
L4 (300 GB/s)	300 GB/s	47 ms	~21
A100 80GB (2.0 TB/s)	2000 GB/s	7.0 ms	~143
H100 SXM (3.35 TB/s)	3350 GB/s	4.2 ms	~239
H200 (4.8 TB/s)	4800 GB/s	2.9 ms	~343

Those are not product numbers. They are the line you cannot cross at batch 1 without shrinking the weights. Measured decode on a real engine is slower: KV traffic, sampling, launch overhead, and the fact that you never hit 100% of advertised bandwidth. If your engine reports 90 tokens/s for 7B-bf16 on an A100 at batch 1, you are at 63% of the floor — a plausible, even healthy, result. If it reports 200 tokens/s, you are measuring the wrong thing or you are not at batch 1.

A 70B model in fp16 is ~140 GB and does not fit on one of those cards. In int8 it is ~70 GB and the A100 floor at batch 1 is 70e9 / 2.0e12 = 35 ms (~29 tok/s). Quantization is not a quality trick here. It is how you buy bandwidth.

Prefill is a different formula

Prefill of s tokens does, very roughly, 2 × N × s FLOPs for the matmuls of an N-parameter dense decoder (the 2 is multiply-add). For 7B and a 4,096-token prompt that is ~5.7e13 FLOPs. On an H100 at 989 TFLOP/s that is a compute floor of ~58 ms, plus attention which FlashAttention keeps closer to compute-bound than the naive s² materialisation would be.

Same model, opposite scarce resource. This is why "the model is slow" is not a sentence you are allowed to say. Slow at what.

Batching, with the catch

At decode, a second sequence is almost free in weight traffic: the weights are already on the bus. KV traffic is not free. Each sequence carries its own cache, and attention over that cache grows with live tokens. The happy region is:

weight traffic  ≫  KV traffic

which holds at modest batch and modest context and then dies. A 32K context at batch 32 can easily make KV the larger term. Then "batching is free" becomes false, and you are back in the memory regime for a different reason.

Write the two terms down before you pick a max batch size:

bytes/step  ≈  W_bytes  +  kv_bytes_per_token × live_tokens

The first term is why batching works. The second is why it stops.

Speculative decoding as regime conversion, with a number

A draft model proposes k tokens. The target verifies them in one forward that looks more like a short prefill than like k decode steps. If the draft is accepted with probability α per token (independent, for a first model), expected accepted tokens per verification is (1 − α^{k+1}) / (1 − α) including the guaranteed first token after a rejection-adjusted view — the practical engineering number is simpler:

If you accept 3 tokens per target forward, and the draft costs 0.2× the target, you have bought a 3 / 1.2 ≈ 2.5× reduction in target forwards.

It only works because those 3 tokens were going to be 3 bandwidth-bound steps. You replaced them with 1 more-compute-bound step plus a cheap draft. If decode were compute-bound, speculative decoding would be much harder to justify.

Mistakes that look like understanding

Reporting tok/s without batch, context, and whether it is prefill or decode. The number is then a press release.

"Bigger GPU" as the decode fix. More SMs with the same HBM generation can raise prefill and do almost nothing for batch-1 decode. Check the bandwidth column, not the TFLOP/s column.

Fitting the model and calling it done. Weights are the fixed cost. KV is the variable cost. The product question is live tokens, not "does 7B fit."

Tuning max_num_seqs by vibes. That knob moves you along the throughput-latency curve and changes who OOMs. Measure TTFT, ITL, and KV bytes together.

In production

A chat SLO is usually two numbers: TTFT under X ms, ITL under Y ms, at a given concurrency. Those map onto prefill and decode. Mixing them into "latency" hides the lever. A RAG system with 8K prompts is a prefill problem that will look like a decode problem if you only watch tok/s.

Disaggregated serving — prefill on one pool, decode on another — is this lesson taken to infrastructure. Different regimes, different hardware, a KV handoff between them. You do not need that architecture on day one. You do need the vocabulary that makes it an option rather than a rumour.

Checkpoint

Explain out loud, without notes, in four sentences:

1. What prefill and decode each do, and which regime each sits in.
2. Why decode is bandwidth-bound.
3. What the KV cache limits, and why it — not the weights — usually caps concurrency.
4. Why bigger batches buy throughput and cost latency.

If you can do that cleanly, you have the vocabulary to hold a real conversation about inference performance. The rest of the course is about making it concrete enough to act on — starting, in the next module, with measuring these effects yourself in PyTorch.

Answers

1. Prefill: process the prompt in parallel, compute-bound (usually). Decode: one token at a time, memory-bandwidth-bound at small batch because you reread the weights.
2. Arithmetic intensity of decode is ~2 FLOP per weight-byte in fp16 — far below any modern ridge. Time is W / B at batch 1.
3. KV grows with layers × KV-heads × head-dim × bytes × live tokens. Weights are paid once. Remaining VRAM divided by bytes/token is concurrency.
4. The weights are already being read. Extra sequences reuse that traffic until KV and latency SLOs intervene. Bigger batches fill the GPU and lengthen queues.

Go deeper

• Pope et al., Efficiently Scaling Transformer Inference — the prefill/decode split made quantitative.
• Practice: KV Cache Capacity Planner, later in module 5. You can attempt the arithmetic now.