Abstract
You call an LLM API. A few seconds later, tokens start streaming back. But what actually happens in between?
In this visually driven talk, we’ll follow a single request from the client to the GPU and back. Through animated visualizations, we’ll explore how the different systems work together, where latency comes from, and the optimizations that make inference faster.
Description
When you call an LLM API, a complex system springs into action. Your request passes through gateways, tokenizers, routers, and schedulers before reaching the GPUs that run the model. Along the way, infrastructure decisions and optimizations determine how quickly the first token arrives and how fast the rest of the response is generated.
But how does all of this actually work?
Rather than explaining LLM inference as a collection of disconnected concepts, this session takes a different approach: we’ll follow a single request, step by step, from API call to GPU execution and back to the client.
Through animated visualizations and practical examples, we’ll explore:
- How the different components of an LLM inference pipeline work together
- How production systems optimize inference performance
- How to understand key performance metrics such as time to first token (TTFT), decode speed, and end-to-end latency
At each stage, we’ll answer three practical questions: What happens here? What slows it down? And which optimizations help, at what cost?
By the end, you’ll understand how production LLM serving systems fit together, where performance bottlenecks arise, and how to reason about improving inference performance.
Key Takeaways
You’ll leave with a practical mental model of LLM inference, an understanding of common performance bottlenecks, and insight into how production inference systems optimize performance. No machine-learning background required.
Audience
Software engineers, backend developers, platform engineers, and AI application developers who want to understand what happens behind an LLM API.
Talk Level
Intermediate
Duration
45 mins
Session Type
talk
Resources
coming soon