Local Model Deployment

You want to run a large language model like the ones behind ChatGPT on your own laptop or desktop, without sending your prompts to a company's servers. This concept gives you the practical skills to do exactly that: you will learn what a model actually is as a file on disk, why it might not fit in your computer's memory, and how to use a tool called Ollama to download, run, and query a model locally. Once you can do this, you unlock the path to deeper topics in your library: understanding quantisation (a process of shrinking a model’s precision so it uses less memory), inference optimization (methods to make the model respond faster), and eventually edge and air-gapped inference for devices where the internet never reaches.

Imagine you've written a long story and want to keep it in your pocket. You could carry the full manuscript, but it's heavy. Or you could condense it to a few pages of summary, losing a little detail, but now it fits easily. A large language model is like that manuscript—a huge file on a server somewhere, containing learned patterns from terabytes of text. When you use ChatGPT, you're not carrying the file; you're sending a request to a computer that has it, and it sends back an answer. Local deployment means copying that file onto your own machine and running the model there.

Models are enormous—a 7-billion-parameter model (a model where 'parameter' refers to the adjustable settings learned during training) takes up several gigabytes of memory. Your computer might have 8, 16, or 32 GB of RAM. If the model doesn't fit, you have two choices: find a smaller model (like a pocket-sized summary) or compress the one you have. Quantisation is that compression; it reduces the number of bits used to represent each part of the model, making it smaller but potentially slightly less precise.

Once it fits, you can run it as a local service. This is a program on your machine that listens for prompts and returns answers, just like a tiny version of ChatGPT that lives inside your computer. To understand why this matters, consider 'batching' (grouping multiple requests together to process them simultaneously) and 'caching' (saving previous results so you don't have to recalculate them). These techniques help the local program use resources efficiently. If you need privacy or work without internet, 'edge inference' refers to running the model on a local device rather than a central server, while 'air-gapped inference' means doing so on a machine that has never been connected to the internet.

Think of ordering food from a restaurant versus cooking at home. Ordering from a restaurant (calling an API, which stands for Application Programming Interface—a set of rules allowing different software programs to talk to each other) is convenient: you pick up the phone, tell them your dish, they cook it in their kitchen with their equipment, and send it to you. You don't worry about ingredients, stove, or chef skill—but you pay per meal, you can't change the recipe, and if the restaurant goes offline, you dine in the dark.

Cooking at home (local deployment) means buying the recipe book (the model weights, which are the actual stored values the model uses to make predictions), having the right pots and pans (enough RAM and a GPU, or Graphics Processing Unit, a type of processor good for parallel calculations), and doing the cooking yourself. The first time is fiddly, but after you've done it, you can cook any time, you customise the recipe, you eat offline, and the marginal cost of each meal is just electricity. The analogy breaks down here: a restaurant can cook dishes you've never seen (the API model is often smarter), and a home kitchen is limited by its utensils (your hardware's memory and speed). So local deployment is a tradeoff: you gain privacy (your data never leaves your home), cost predictability (no per-token billing), and offline use, but you lose the capability ceiling and you take on the responsibility of maintaining your own 'kitchen'—installing software, ensuring enough memory, and troubleshooting when something breaks.

Local model deployment is the act of installing a trained machine learning model onto your own hardware (a laptop, desktop, or single-board computer) and running it there, so that you can send prompts and receive outputs entirely without internet access or a third-party API.

This concept sits at the point where machine learning meets practical, everyday software. In your library's structure, it belongs to Inference Optimization—the parent topic that covers quantisation, distillation, batching, and caching. You have notes on Quantization, which is the most immediate tool for making a model fit locally; Edge and Air-Gapped Inference, which extends local deployment to devices that must never touch the internet; and Latency and Speedups, which becomes relevant when you want your local model to feel fast.