Occupancy is not the goal
Three kernels with occupancy, bytes, and time. Decide whether raising occupancy is the lever, a sideshow, or active harm.
- Module
- The GPU execution model
- Objective
- Treat occupancy as a latency-hiding condition, not a score, and connect it to the regime the kernel is actually in.
Occupancy is "how many warps can live on the SM." It is not "how fast is the kernel." This drill is that distinction, with numbers.
Part A — Definitions, from memory
Answer without the lesson:
- Warp size on NVIDIA.
- What occupancy is a fraction of.
- Two resources that cap occupancy.
- Why a memory-bound kernel likes occupancy more than a math-heavy tensor-core kernel might.
Part B — Three kernels
Card: 108 SMs, 2.0 TB/s, 312 TFLOP/s fp16 tensor.
| kernel | occupancy | achieved GB/s | achieved TFLOP/s | notes |
|---|---|---|---|---|
| A streamed add | 18% | 1.82e3 | 0.3 | 1 FLOP/byte |
| B streamed add | 72% | 1.91e3 | 0.3 | same bytes |
| C large GEMM | 34% | 0.9e3 | 240 | tensor cores, register-heavy |
- A vs B: did occupancy win? How many percent of wall time could it possibly still win?
- Someone wants to spill registers on C to raise occupancy to 70%. What extra traffic risk are they buying, and is C compute-bound enough that you refuse?
- Write the question from the lesson: is the resource limiting occupancy also the resource dominating runtime?
Part C — Launch too small
A decode kernel uses 128 threads (4 warps) per block, 8 blocks total, on 108 SMs.
- How many SMs are idle by construction?
- Occupancy on the busy SMs could be 100%. Why is that a trap metric here?
- The serving lever that actually fills the machine?
Acceptance
- Warp 32. Occupancy ⊂ resident warps / max warps per SM. Registers and smem cap it.
- A→B: bandwidth already 91% of peak; occupancy is done. C: do not dump registers if you are already at 77% of tensor peak.
- 8 blocks on 108 SMs is an occupancy-of-the-wrong-thing story: the GPU is unfilled. Batch.
Stretch
Look up one Nsight Compute occupancy pie for any kernel you have. Write four lines: limiter, regime, whether you will touch occupancy, next experiment.
Check
B's extra occupancy bought ~5% bandwidth on an already-saturated bus. C is compute-bound; extra occupancy via spilling can increase HBM traffic and slow it down. Tiny grid: occupancy is a vanity metric.
Debrief
Ask "what am I hiding latency from?" If the answer is "HBM" and you are already at peak GB/s, stop. If the answer is "there is no work on the SM," occupancy was never the name of the problem — launch geometry and batch were.