Practice Ladder

The course teaches the model. The ladder makes it usable. Each problem asks for an artifact an inference engineer would actually produce: a benchmark harness, a profiler report, a capacity plan, a scheduler simulation, or an incident playbook.

Do them with a timer, but do not optimise for speed. Optimise for a checkable chain of reasoning: prediction, measurement, explanation, and the next action.

M01.01

Accounting

Tensor Memory Accountant

Given tensor shapes, dtypes, and view/copy operations, produce a memory ledger that separates storage, traffic, and accidental copies.

PyTorch and the memory wall

Warmup·35 minutes

M01.02

Benchmarking

Honest GPU Timer

Build a tiny benchmarking harness that warms up, uses CUDA events, repeats, reports medians, and refuses to benchmark CPU fallbacks by accident.

PyTorch and the memory wall

Core·45 minutes

M01.03

Accounting

Roofline Classifier

Classify toy kernels as memory-bound, compute-bound, or overhead-bound from flops, bytes, time, and hardware ceilings.

PyTorch and the memory wall

Core·40 minutes

M02.01

Profiling

Profiler Triage Report

Turn an operator table into a concise performance report with suspects, evidence, and the next experiment.

See inside the box

Core·50 minutes

M02.02

Profiling

Timeline Idle Gap Detective

Given a toy timeline, identify whether the GPU is waiting on launch overhead, host work, synchronization, or data transfer.

See inside the box

Hard·60 minutes

M05.01

Serving

KV Cache Capacity Planner

Compute live-token capacity for an LLM deployment and turn VRAM into max concurrency under different prompt and output lengths.

Serving an LLM

Core·50 minutes

M05.02

Systems

Continuous Batching Simulator

Write a small scheduler simulation that admits new requests between decode steps and compares it to static batching.

Serving an LLM

Hard·90 minutes

M05.03

Operations

Serving SLO Playbook

Design the measurement and rollout plan for an inference endpoint with real latency targets, capacity limits, and rollback rules.

Serving an LLM

Capstone·2 hours