Every LLM request has two phases, and they are bottlenecked by completely different things. Scene 1 in the lab walks one request through both.
Prefill reads your whole prompt at once. All the tokens go through the model in parallel, so it is a compute-heavy burst. It decides time to first token (TTFT).
Decode then writes the answer one token at a time. For every single token, the GPU re-reads all of the model's weights from memory. That makes decode memory-bandwidth bound, and it decides inter-token latency (ITL): how fast the words stream.
Rough numbers
From my model of Llama 3.1 8B on one H100:
- Reading 16 GB of weights at 3.35 TB/s takes about 4.8 ms. That is the floor for every decode step, even for a single user.
- Prefilling a 2,000-token prompt takes about 50 ms of compute at the throughput I assumed (~600 TFLOPS achieved).
Why it matters
The fixes are different. Long prompts and slow first tokens point at prefill: prefix caching, chunking, more compute. Slow streaming points at decode: batching, quantization, speculative decoding, more memory bandwidth.
If a team reports "latency" as one number, ask which one.
Next: why batching is the biggest throughput lever in decode.