Reading the GPU timeline

The PyTorch profiler tells you which operators matter. A timeline tells you whether the machine is being fed.

This matters because many inference bottlenecks are not inside a kernel. They are between kernels: launch gaps, CPU work, synchronization, data movement, and scheduling decisions that keep the GPU empty.

What a healthy timeline looks like

A healthy GPU timeline is boring. CPU ranges launch work, GPU lanes fill with kernels, memory copies are either absent or intentionally overlapped, and the critical path has very little blank space.

A suspicious timeline has rhythm:

CPU:  prepare | launch | wait | prepare | launch | wait | prepare | launch | wait
GPU:           kernel          kernel          kernel          kernel

That pattern says the GPU is fast enough to finish before the host can feed it. Optimising the kernel may make the workload slower from the user's point of view, because it makes the blank space larger as a share of total time.

Label the program

Nsight Systems becomes much more useful when the trace has named ranges. In Python, NVTX ranges let you mark semantic regions:

import torch.cuda.nvtx as nvtx

nvtx.range_push("prefill")
prefill()
nvtx.range_pop()

nvtx.range_push("decode")
for _ in range(max_new_tokens):
    nvtx.range_push("decode_step")
    decode_one_token()
    nvtx.range_pop()
nvtx.range_pop()

Now the CPU timeline says what the program was trying to do, and the GPU timeline shows what actually ran underneath those ranges.

The important move is comparison:

Does the GPU work begin quickly after the range begins, and does it stay dense until the range ends?

If no, the bottleneck may live outside the kernel.

Four timeline smells

Launch picket fence. Many tiny kernels separated by visible gaps. This often points to overhead-bound work. Candidate levers: fusion, larger batches, torch.compile, CUDA graphs, fewer Python-loop iterations, or a serving engine that batches decode steps across requests.

Host island. A wide CPU region with no GPU activity. Candidate levers: move preprocessing out of the request path, parallelise CPU work, cache tokenization, stop logging per token, or avoid a synchronization that forces the host to wait.

Copy on the critical path. Host-to-device or device-to-host transfer sits between the request and the next kernel. Candidate levers: keep tensors on device, use pinned memory for necessary transfers, overlap copies when possible, or remove an accidental .cpu(), .item(), or NumPy conversion.

Sync cliff. The CPU blocks until the GPU catches up. Some synchronization is intentional; hidden synchronization is not. In PyTorch, common culprits include .item(), printing CUDA tensors, measuring without events, and code paths that need a scalar result on the host.

Timeline before cure

The timeline does not tell you the fix by itself. It tells you what kind of fix is allowed.

If the GPU lane is dense and a kernel owns the run, kernel-level work can matter. If the GPU lane has holes, the first job is feeding the machine. If host-to-device copies dominate, changing a matmul kernel will not save you. If decode is a picket fence of tiny steps, a better serving scheduler may beat a lower-level kernel rewrite.

That is why timeline literacy is part of inference engineering and not just performance tooling. It keeps the level of intervention honest.

A trace-reading protocol

Use the same protocol every time:

1. Name the workload: model, batch, prompt length, output length, dtype, hardware.
2. Mark the phases: prefill, decode, postprocess, transfer, tokenize if relevant.
3. Zoom out: is the GPU mostly dense or mostly waiting?
4. Zoom in: choose the largest blank region on the critical path.
5. Explain the blank region using visible CPU work, copy, sync, or launch overhead.
6. Propose one experiment that would make the blank region shrink.

If step five is guesswork, collect a better trace. If step six has more than one experiment, pick the one closest to the blank region.

References

• NVIDIA Nsight Systems user guide
• NVIDIA Nsight Systems overview

Checkpoint

1. What does a dense GPU lane tell you that a top-operators table cannot?
2. Name three PyTorch actions that can accidentally synchronize the CPU with the GPU.
3. A decode trace shows hundreds of tiny kernels with gaps between them. What level of fix would you try before writing a custom kernel?