Serving SLO Playbook

Prompt

You own a new LLM endpoint. Product wants:

P50 time to first token under 600 ms
P95 time to first token under 1.8 s
P95 end-to-end latency under 12 s for 1K output tokens
error rate under 0.5%
no OOM during normal traffic

Traffic has three request classes:

1. Chat: short prompt, short output, latency sensitive.
2. Analysis: medium prompt, long output, throughput sensitive.
3. Document QA: long prompt, medium output, memory sensitive.

Deliverable

Write a launch playbook with:

1. Load-test matrix: prompt length, output length, concurrency, request mix.
2. Metrics: latency, throughput, queueing, live tokens, KV memory, GPU utilization, error modes.
3. Dashboards: the five charts you would watch during launch.
4. Admission controls: limits that prevent one workload from harming the others.
5. Rollback rules: exact thresholds that trigger rollback.
6. Follow-up experiments: what to tune after the endpoint is stable.

Constraints

1. You cannot buy more GPUs for the first launch.
2. You can tune scheduler limits.
3. You can split traffic by request class.
4. You can reduce max context for one class if the product accepts it.

Acceptance criteria

A strong playbook:

1. Uses percentiles instead of averages for latency.
2. Separates queue wait from model execution.
3. Treats live tokens and KV memory as first-class capacity metrics.
4. Has rollback rules specific enough that an on-call engineer does not need interpretation.
5. Does not optimize throughput by silently breaking chat latency.

Stretch

Add a "brownout" plan: what features or request classes degrade first when traffic exceeds capacity?

Debrief

Inference engineering does not end at a benchmark. The endpoint only counts when it can be operated under load without guessing.