Latency and Speedups

You are reading this because you want to build or use AI systems that feel fast. If you press a button in an app and the response takes five seconds, you'll blame the app. But most of that time is not the model 'thinking' — it's the way the system moves data around. This concept teaches you how to make a model feel faster without making the model itself smaller or smarter. It unlocks the rest of Inference Optimization: quantization (making the model fit in less memory), distillation (training a tiny student to copy a large teacher), and local deployment (running the model on your own laptop instead of a cloud API). Each of those topics assumes you already understand the tricks in this page: caching, batching, streaming, and decoding strategy. Master this and you'll never again accept a slow AI system as inevitable — you'll know exactly which lever to pull.

Imagine you run a small tea stall near a railway station. The stall has one kettle, and the kettle takes exactly two minutes to boil. You have a queue of customers asking for chai. Each customer asks for the same type of chai. You could boil the kettle fresh for each customer — two minutes each. But that's slow. So you boil a full kettle once, and every customer who asks for the same chai gets a cup instantly. That's caching: you remembered the result and reuse it instead of recomputing. But what if every customer asks for a different chai — some with ginger, some with less sugar? The kettle still takes two minutes, but you can put water for several cups in the same kettle at the same time. Instead of twenty minutes for twenty customers, you boil one large kettle in four minutes and pour twenty cups. That's batching: you combined several jobs into one, taking longer per job but far less time per job. Now, what about the customer who is standing there, waiting? They don't care that the total order will be ready in four minutes; they want something to drink now. You give them a half-filled cup as soon as the water is hot, then refill as soon as the next bit is ready. They start sipping in thirty seconds, even though the full cup takes four minutes. That's streaming: you send the output as it's produced, not all at once. Finally, the 'decoding strategy' is the order you pour the tea. If a customer says 'three chai, two with ginger, one without,' you don't need to boil the kettle and wait for the third chai before you start pouring the first. You start pouring the first one immediately — the model can start generating the first few words while the rest of the sentence is still being decided. All four tricks — caching, batching, streaming, decoding — attack the same enemy: time spent waiting.

Think of a large language model as a librarian who writes a book one page at a time. You ask the librarian: 'Write a one-page summary of the causes of the 1857 uprising.' The librarian must read a huge reference shelf to answer. Without caching, the librarian would re-read the entire shelf for every page they write. With caching, they read the shelf once and keep their notes handy — that's the KV cache in a transformer. The typing speed (the model's generation) is fixed, but the flurry of re-reading is eliminated, so the first page appears almost instantly. Now, suppose twenty readers all ask slightly different summaries on the same shelf. The librarian could read separately for each — twenty times the shelf reading. Or they could gather all the readers, read the shelf once, and then write all twenty summaries in sequence — the total time might be longer for the first reader, but everyone gets their answer sooner. That's batching. But here's the catch: if the librarian types the entire summary for the first reader before starting the second, the first reader sees a long pause — the page with the title appears, then nothing for a minute. Instead, the librarian types the first line of summary A, then the first line of summary B, then the second line of A, and so on. Everyone sees a line appear every few seconds, even though the last line arrives at the same time as it would have with no batching. Perceived latency drops while throughput stays high. Finally, the decoding strategy: the librarian doesn't have to decide the final word of a sentence before writing the first word. They can start 'In 1857…' and decide the rest while the reader is already reading. With greedy decoding, they choose the most likely next word and move on. With beam search, they keep several possible continuations in mind, but that costs time. The choice of decoding strategy is a trade-off: how much time you spend trying to be clever versus just getting the next word out fast.

Latency is the time from sending a request to receiving the complete response; speedups are techniques that reduce that time — either by making the arithmetic faster, or by making the response appear faster than it really is (perceived latency), through caching, batching, streaming, and clever decoding choices.

You have not learned any other topic in this library yet — this is your first page. But your library records a set of topics that this one builds toward: Edge and Air-Gapped Inference, Knowledge Distillation, Local Model Deployment, Quantization, and Test-Time Compute. Each of those assumes you understand the latency techniques in this page. Quantization, for instance, makes a model smaller so it fits on a phone — but the phone still needs streaming to feel responsive. Distillation creates a small student model — but a small model's speedup is useless if the serving framework doesn't batch requests. So think of this page as the foundation of the entire Inference Optimization folder.