MGMilind Gunjan / gunjanlabs

LLM inference, explained · Part 5 of 5 · Nov 22, 2026 · 1 min read

Putting it together

Four ideas, one chain. How they fit on one GPU, what each optimization pulls on, and what to actually watch.

Four posts, four ideas. Here is how they fit together on one GPU, and what I'd actually watch.

The chain

  • Decode re-reads the weights every token, so it is memory-bandwidth bound.
  • Batching shares that read across users, so throughput comes from batch size.
  • Batch size is capped by KV cache memory.
  • Prefill bursts compete with decode for the same GPU, so long prompts stall streaming for everyone.

Every optimization pulls on one link

Run the "Try this" sequence in the whole-GPU scene to see each one:

  • Prefix caching: a shared system prompt is prefilled and stored once. Time to first token drops from seconds to tens of milliseconds.
  • FP8 KV cache: half the memory per token, so roughly twice the users fit.
  • Chunked or disaggregated prefill: prompts stop stalling decode. Disaggregation moves them to a separate pool and adds a KV transfer.
  • Speculative decoding (scene 3): a small model drafts tokens, the big model checks several in one pass. At 75% acceptance with 4 drafts that is about 3 tokens per big-model pass, with identical output.

What to watch instead of utilization

Queue depth, KV memory used, time to first token, inter-token latency, and goodput: the share of requests that meet your latency target.

Why this matters in telecom

Operators are starting to put GPUs at edge sites to run inference closer to users and devices (NVIDIA on AI grids). There, the latency budget is round trip plus queueing plus prefill plus decode, and every idea in this series spends part of it.

The simulation teaches the shape. The next step is measuring a real serving endpoint against it.

Illustrative model built with AI assistance; assumptions and numbers checked by me. Not a benchmark. See this in the lab →

Open the full lab →