Writing
LLM inference, explained
A five-part series on what actually happens on an inference GPU, and what to watch instead of utilization. Each part pairs with a live scene in the inference lab.
- Part 1GPU utilization lies
Your dashboard says 100%. Your users say the chatbot takes four seconds to answer. Both are right.
- Part 2Prefill vs. decode
Every LLM request has two phases, and they are bottlenecked by completely different things.
- Part 3Continuous batching
Decode reads the weights once per step, no matter how many users share it. That is the good news hidden in the bottleneck.
- Part 4KV cache memory sets concurrency
If batching is where throughput comes from, memory decides how big the batch can get.
- Part 5Putting it together
Four ideas, one chain. How they fit on one GPU, what each optimization pulls on, and what to actually watch.
Illustrative model built with AI assistance; assumptions and numbers checked by me. Not a benchmark.
Earlier writing on Medium
Hands-on notes from the lab and the field, 2019 onward. These open on Medium.
Telco & AI-RAN
- Sep 2025
- Sep 2025
- Nov 2023
- Nov 2023
- May 2023
- Feb 2023
- Apr 2022
Cloud-native, security & infrastructure
- Sep 2025
- Jun 2023
- Apr 2023
- Dec 2022
- Dec 2022
- Apr 2019