Quantization

Quantization is the key that lets a model that needs gigabytes of memory run on your laptop or phone. Without it, large language models are locked to expensive cloud servers. This unlocks: Local Model Deployment (run models offline and privately), Edge and Air-Gapped Inference (serve on devices with no internet), and makes all of Inference Optimization cheaper. Quantization is also the reason you can download a 7-billion-parameter model with Ollama and get responses in seconds on a consumer laptop.

A computer stores numbers in chunks called bits. A 32-bit number can represent billions of different values. But do we always need that much precision? Think of weighing rice at a shop. The shopkeeper can use a digital scale that shows grams to three decimal places, or a rough scale that shows to the nearest 10 grams. For most purposes, the rough scale is fine and much cheaper. Similarly, a model's weights are just numbers that tell it how strongly to listen to each input. If we round those numbers from 32-bit precision down to 8-bit, we still capture the essential spread, just with less detail. The model's output changes slightly—like the rice being a few grams off—but the meaning stays the same. Quantization is exactly this rounding, done at scale, across millions of numbers, to save memory and speed things up.

Imagine you are giving directions to a friend over a phone that has a very weak signal. You can either send every detail—every tiny turn, every bump in the road—and have it take a long time and drain battery, or simplify. You decide to round distances: instead of "walk 1.03 km then turn right after 0.47 km", you say "walk 1 km, turn right, walk half a km". The friend still gets there, maybe a few meters off, but the message is shorter and faster. In the model, the weights are the directions. Quantization rounds them to fewer digits. The directions still work, but if you round too much—say, to the nearest kilometer—your friend might walk into a wall. This is the limit: round too coarsely and the model's answers become nonsense. But the sweet spot—8-bit, or even 4-bit—keeps most of the quality while cutting memory by 75% or more. The analogy breaks down because a model has billions of weights, so even small rounding errors can accumulate. But the principle is the same: we trade a little accuracy for a lot of savings.

Quantization is representing each weight and activation in a model using fewer bits than the usual 32-bit floating point, reducing the model's memory footprint and often speeding up computation, at the cost of a small, measurable degradation in output quality.

You are learning Inference Optimization. Quantization is one of the main tools in that toolbox, alongside Knowledge Distillation (which shrinks the model by training a smaller copy) and Local Model Deployment (where quantized models are loaded into Ollama and run on a laptop). All of these make serving models cheaper and faster. Quantization specifically is why you can run a 7B model on 8GB RAM: the file is roughly a quarter the size.