MGMilind Gunjan / gunjanlabs

LLM inference, explained · Part 3 of 5 · Nov 22, 2026 · 1 min read

Continuous batching

Decode reads the weights once per step, no matter how many users share it. That is the good news hidden in the bottleneck.

Last post: every decode step re-reads all of the model's weights. Here is the good news hidden in that.

The weights are read once per step no matter how many users are in the batch. Add more users and each step only gets slightly slower, because the extra cost is just their own KV cache.

The arithmetic

In my model of Llama 3.1 8B on one H100:

  • 1 user: about 4.8 ms per step, so 1 token every 4.8 ms.
  • 32 users with ~2,000 tokens of context each: about 7.3 ms per step, so 32 tokens every 7.3 ms.

That is roughly 21× the throughput for about 1.5× the step time. Batching is where inference throughput comes from.

How you fill the batch matters

Scene 2 in the lab runs the same requests on two schedulers:

  • Static batching waits for the longest answer in the batch to finish before refilling. Short answers leave slots idle.
  • Continuous batching refills a slot the step after it frees up.

Every idle slot is a share of a weight read you already paid for and threw away. That is why continuous batching is the default in vLLM, SGLang and TensorRT-LLM.

The limit on batch size isn't compute. It's memory, which is the next post.

Illustrative model built with AI assistance; assumptions and numbers checked by me. Not a benchmark. See this in the lab →

Next: KV cache memory sets concurrency →