Inference practice
M01.06·Accounting·Core·90 minutes·3 min read

Decode intensity from scratch

Derive arithmetic intensity of a decode step from parameter count and dtype, then place it on four ridges without looking at a blog.

Module
PyTorch and the memory wall
Objective
Compute decode intensity from first principles and show it sits below every modern ridge at batch 1.

No profiler. A 7B dense decoder, batch 1, one new token, ignore KV at first.

Part A — FLOPs and bytes

A linear layer y = x @ W with W of shape (in, out):

  • FLOPs ≈ 2 × in × out (macs)
  • Bytes ≈ in×out × elem to read W, plus tiny x and y

For a decode step, x is 1×hidden. Weight traffic dominates.

  1. Why is total decode FLOPs ≈ 2 N for N parameters (dense, one token)?
  2. Why is weight traffic ≈ N × bytes_per_param?
  3. Intensity in FLOP/byte for fp16, bf16, int8, int4?

Part B — Place on ridges

Ridges: T4 203, L4 807, A100 153, H100 295 FLOP/byte (from the ridge workbook).

Mark decode batch 1 for each dtype below / above each ridge.

Then: at what batch size would fp16 decode intensity cross the A100 ridge if KV were still free? (It is not free. Compute the fake number anyway, then say why it is fake.)

Part C — Prefill contrast

Prefill of s tokens: FLOPs ≈ 2 N s for the matmuls. Traffic is still about one read of W, plus activations.

  1. Intensity vs s, ignoring activation traffic.
  2. At what s does intensity cross 156 FLOP/byte?
  3. One sentence: why prefill and decode disagree about "is this model compute-bound?"

Part D — KV ruins the batching fairy tale

7B, 32 layers, 8 kv heads, 128 dim, bf16 → 128 KiB/token (you will prove this in module 5; use it here).

At batch 32, 4K live tokens per sequence: KV bytes in memory? Extra traffic per step if you read all of it? Does weight-only intensity still describe the step?

Acceptance

  1. fp16 decode intensity = 1 FLOP/byte. int8 = 2. int4 = 4.
  2. All four sit below all four ridges at batch 1.
  3. Crossing A100's 153 needs a fictional batch ~153 if KV is ignored — and KV is not ignorable there.
  4. Prefill intensity scales with s; decode does not.

Check

A. Each parameter is used once per token in the big GEMMs: 2 FLOPs (mul+add) per param. You must ship the param from HBM: 2 bytes in fp16. Intensity 1.

B. 1, 2, 4 all << 153. Batching without KV: intensity ≈ batch, so batch 153 "crosses" A100. Fake because KV traffic grows with batch × context.

C. Intensity ≈ 2 N s / (2 N) = s FLOP/byte in the weight-only model. Cross 156 around s≈156 tokens. Real prefill is messier (attention) but the direction is the lesson: long prompts are compute-ish; single-token decode is not.

D. 32 seq × 4096 × 128 KiB ≈ 16 GiB KV. You are no longer in "just read 14 GB of weights."

Debrief

This is module 0's decode floor, derived from module 1's intensity. If the two lessons do not click into one number, do this drill again before fusion.