Last post: every decode step re-reads all of the model's weights. Here is the good news hidden in that.
The weights are read once per step no matter how many users are in the batch. Add more users and each step only gets slightly slower, because the extra cost is just their own KV cache.
The arithmetic
In my model of Llama 3.1 8B on one H100:
- 1 user: about 4.8 ms per step, so 1 token every 4.8 ms.
- 32 users with ~2,000 tokens of context each: about 7.3 ms per step, so 32 tokens every 7.3 ms.
That is roughly 21× the throughput for about 1.5× the step time. Batching is where inference throughput comes from.
How you fill the batch matters
Scene 2 in the lab runs the same requests on two schedulers:
- Static batching waits for the longest answer in the batch to finish before refilling. Short answers leave slots idle.
- Continuous batching refills a slot the step after it frees up.
Every idle slot is a share of a weight read you already paid for and threw away. That is why continuous batching is the default in vLLM, SGLang and TensorRT-LLM.
The limit on batch size isn't compute. It's memory, which is the next post.