MGMilind Gunjan / gunjanlabs

LLM inference, explained · Part 4 of 5 · Nov 22, 2026 · 1 min read

KV cache memory sets concurrency

If batching is where throughput comes from, memory decides how big the batch can get.

If batching is where throughput comes from, memory decides how big the batch can get.

GPU memory gets split two ways. The model weights load once. Then every concurrent user needs their own KV cache: the keys and values for every token in their conversation, so the model never has to recompute them.

KV per token is simple arithmetic

2 × layers × KV heads × head size × bytes. For Llama 3.1 8B in FP16:

2 × 32 × 8 × 128 × 2 = 128 KiB per token.

On one 80 GB H100

Scene 4 in the lab lets you change each of these:

  • Weights take 16 GB, and I allow ~2 GB for runtime overhead, leaving ~62 GB.
  • At 8k tokens of context, each user's KV cache is about 1.07 GB.
  • So about 57 users fit at once. Store the KV cache in FP8 and it is about 115.
  • Switch to Llama 3.1 70B with FP8 weights and only ~8 GB is left: room for about 2 users at 8k. That is why bigger models get split across GPUs.

When memory runs out, the server stops admitting new requests, and under pressure it preempts running ones and recomputes them later. Users feel that as a stalled answer.

The levers

A shorter maximum context, an FP8 KV cache, prefix caching so a shared system prompt is stored once, or more memory per replica.

Illustrative model built with AI assistance; assumptions and numbers checked by me. Not a benchmark. See this in the lab →

Next: Putting it together →