The GPU execution model without mysticism

A GPU is not magic parallel dust. It is a machine with a small number of rules, repeated at brutal scale.

The purpose of this module is not to make you a CUDA specialist yet. It is to give you enough of the execution model that profiler output stops looking like folklore.

The hierarchy

At a simplified level:

1. A GPU has many streaming multiprocessors, usually called SMs.
2. A kernel launch creates a grid of thread blocks.
3. Blocks are assigned to SMs.
4. Threads inside a block execute in groups called warps.

On NVIDIA GPUs, a warp is 32 threads. When people say "the GPU likes parallelism," they mean the hardware wants enough active warps to hide latency while memory and arithmetic pipelines are waiting.

If you launch too little work, the machine is underfilled. If each thread uses too many registers or each block uses too much shared memory, fewer blocks can reside on an SM at once. That resident-work limit is part of occupancy.

High occupancy is not the goal. It is a condition that may help the GPU hide latency. A kernel can have high occupancy and still be memory-bound. A kernel can have lower occupancy and still be fast if each warp does a lot of useful work.

The memory ladder

The fastest memory is closest to the thread and smallest:

registers       fastest, per thread
shared memory   fast, per block
L2 cache         shared across SMs
HBM / global     huge, high bandwidth, high latency
host memory      off device, usually not on the critical path

Inference work often lives or dies by this ladder. The same arithmetic can be cheap or expensive depending on how many times it pulls from global memory.

That is why earlier lessons kept asking for bytes moved. The GPU execution model is where those bytes become concrete: global memory reads are not abstract cost, they are long-latency transactions that the hardware tries to hide with other ready warps.

Coalescing

Warps issue memory operations together. If neighboring threads read neighboring addresses, the hardware can combine those reads efficiently. If neighboring threads stride through memory or scatter randomly, the hardware may fetch many memory segments to serve the same warp.

This is coalescing.

A transpose lesson now comes back with teeth. A transposed tensor may be a free view, but a later kernel that walks it in the wrong order can turn clean contiguous access into strided access. The view did not move bytes at creation time; it changed the address pattern for the next kernel.

That is not a reason to call .contiguous() everywhere. It is a reason to measure the consumer.

Occupancy versus arithmetic intensity

Arithmetic intensity says how much math you do per byte moved. Occupancy says how much work can sit on the SM at once. They are related, but not interchangeable.

A low-intensity kernel often needs many active warps to hide memory latency. A high-intensity kernel may spend more time in math pipelines and less time waiting for HBM. A register-heavy kernel may lower occupancy, but that can be a win if the registers avoid extra global memory traffic.

The question is never "is occupancy high?" The useful question is:

Is the resource limiting occupancy also the resource responsible for the runtime?

If occupancy is low because of register use, but runtime is dominated by global memory traffic, lowering registers might not help. If occupancy is low and the timeline shows memory latency stalls, then resource pressure deserves attention.

How this changes PyTorch work

Even if you never write CUDA, this model changes how you interpret high-level code:

1. Many small ops can be slow because each op launches a kernel and underfills the GPU.
2. Reductions can be slow because they require communication and often multiple stages.
3. Indexing and gathers can be slow because memory access is not coalesced.
4. Fusing ops can help because intermediate tensors stop round-tripping through global memory.
5. Bigger batches can help until they hit memory capacity or latency targets.

This is the bridge from PyTorch to kernels. You do not need to see the C++ to reason about whether the kernel is starved, memory-bound, or doing enough work to matter.

References

• CUDA C++ Programming Guide
• CUDA C++ Best Practices Guide

Checkpoint

1. Why can high occupancy be helpful without being the optimisation target?
2. What memory pattern does coalescing reward?
3. A kernel reads random rows from a large embedding table. Which part of the execution model should you suspect first?