Your GPU dashboard says 100% utilization. Your users say the chatbot takes four seconds to answer. Both are right.
I built a small simulation of one H100 serving Llama 3.1 8B to show why. Open it in the lab, leave the defaults running, then raise arrivals to 15 requests per second. Time to first token climbs from milliseconds into seconds, KV memory fills up, requests start getting preempted, and "utilization" never moves off 100%.
Utilization only tells you the GPU is doing something. It says nothing about whether users are waiting.
Five ideas explain almost everything
- Decode re-reads all the model weights for every token, so it is limited by memory bandwidth, not compute.
- Batching lets many users share that one read, which is where throughput comes from.
- Every user's KV cache grows in the same GPU memory, and that memory decides how many users fit.
- Prefill reads a whole prompt in one compute-heavy burst, and it stalls everyone else unless it is chunked or moved.
- Cost is GPU-hours divided by tokens served.
What to watch instead
Queue depth, KV memory used, time to first token, and inter-token latency. The next posts take each idea in turn, each with its own scene in the lab, and the last one ties them together.