Overhead-bound collapse

You fused a four-op pointwise chain. Predicted memory-bound speedup ≈ 4×. Measured:

elements	eager µs	fused µs	ratio
256	11.2	10.6	1.06
4,096	12.0	11.1	1.08
65,536	18.4	12.9	1.43
1,048,576	42	16	2.63
16,777,216	410	108	3.80

Launch overhead ≈ 8–12 µs on this machine.

Part A — Annotate the table

For each row: estimated traffic-bound time at 1.5 TB/s for fused (2× fp16 bytes) vs eager (assume 8×). Compare to 10 µs launch. Label overhead / mixed / memory.

Part B — The paragraph

Explain the shape: why the ratio is ~1 at the top, largest at the bottom, and not a step function. Use launches and bytes only.

Part C — Three levers

For the 256-element row, rank these as useful / useless:

1. A hand-written Triton kernel of the same chain
2. torch.compile
3. CUDA graphs over a 128-token decode loop of this chain
4. Raising batch until elements ≥ 1e6
5. Switching fp16 → fp32

Part D — Decode connection

Decode launches this chain (or worse, dozens of tiny kernels) once per token. Why does "fusion did nothing at n=256" still matter for a 7B server, and what is the serving name of lever 3–4?

Acceptance

1. Top two rows: launch dominates both; fusion cannot 4× a 10 µs tax.
2. Bottom row: traffic ratio appears.
3. Triton at n=256 is useless as a 4×; graphs/batch are the overhead levers.
4. fp32 at overhead-bound sizes makes traffic worse when you finally leave the regime.

Check

256 × 2 bytes × 2 (fused) = 1 KiB. At 1.5 TB/s ≈ 0.7 ns. Entirely launch. 16e6 elements × 2 × 2 = 64 MiB fused ≈ 43 µs floor at 1.5 TB/s; measured 108 µs is same order (plus extra reads). CUDA graphs and continuous batching exist because decode lives in the top rows unless you batch.

Debrief

Module 1 ends when you can look at a sweep and say which axis you are on. If you still say "we should fuse it" at 256 elements, do this table again.