Inference Engineering
A course and practice ladder on why models are slow and what the levers actually are — from counting the bytes a tensor moves, up through profiling, GPU execution, KV-cache math, batching, and serving. It ramps deliberately: nothing in a later module is assumed earlier.
Practical modules are notebooks or drills. Run the notebooks in Google Colab on the free GPU tier, work the drills in a local file, and keep a log of prediction against measurement. The holy grail here is not reading — it is calibration.
In Colab
Download a lesson's notebook, then in Colab choose File ▸ Upload notebook. Set Runtime ▸ Change runtime type ▸ GPU before you run anything, or every measurement will be a CPU measurement.
PyTorch is already installed there. The free tier is enough for every module in this course.
Locally in Jupyter
If you have an NVIDIA card, run it at home and the numbers get more stable — no shared tenancy, no session limits.
pip install jupyterlab pip install torch --index-url \ https://download.pytorch.org/whl/cu124 jupyter lab
5 of 7 modules are open, with 11 lessons and 8 practice problems. The rest are listed so you can see where this goes — the order is the argument, and each one assumes the one before it.
Orientation
The one mental model the rest of the course hangs off: every workload is waiting on arithmetic, on bytes, or on the launch queue, and optimising the wrong one buys nothing. No GPU needed.
PyTorch and the memory wall
Make the model measurable. Count the bytes a tensor really moves, time a GPU honestly, find the line where a kernel stops being limited by maths and starts being limited by memory, and watch fusion move it.
See inside the box
Stop guessing where the time goes. torch.profiler for operator-level timing, then Nsight Systems for the timeline, and learning to read the idle gaps — launch overhead, sync points, transfers.
The GPU execution model
Why kernels are fast or slow, before writing any. SMs, warps and occupancy; the latency and bandwidth gaps between registers, shared memory, L2 and HBM; coalescing and bank conflicts.
Writing kernels in Triton
PlannedThe first kernels, without the C++ tax. Vector add, then fused softmax — the first kernel that beats PyTorch and where you can see why in the memory traffic — then tiled matmul.
Serving an LLM
Where the whole model pays off: prefill against decode, what the KV cache actually caps, continuous batching, and finding the knee of the throughput-versus-latency curve yourself.
Compilers and going lower
Plannedtorch.compile and reading the Triton it generates, CUDA graphs and launch overhead, then C++ and CUDA proper for anyone who wants the specialist path.
The ladder that makes it stick
Each problem produces an artifact: a byte ledger, a benchmark harness, a profiler report, a scheduler simulation, or an operations playbook. That is the difference between reading about inference and becoming useful under pressure.
View every practice problemCurrent set
8
drills across accounting, benchmarking, profiling, serving, systems, and operations.
Predict before you measure
Every notebook marks the places to commit a number before running the cell. Write the number and one sentence of reasoning, then measure. The gap between the two is the actual lesson, and it is invisible if you skip the prediction — you will read the result, nod, and learn nothing.
Keep a running log of predicted against actual. When the gaps start closing, that is direct evidence the model in your head matches the machine.
Explain it in bytes
“It got faster” is not an explanation. “It moves a third of the memory it used to, and the measured speedup matches that ratio to within the launch overhead” is. The second one is checkable, and it generalises to the next problem.
Every module ends with a checkpoint you should be able to answer out loud, without notes, before moving on.