Local Model Deployment

You want to run a large language model like the ones behind ChatGPT on your own laptop or desktop, without sending your prompts to a company's servers. This concept gives you the practical skills to do exactly that: you will learn what a model actually is as a file on disk, why it might not fit in your computer's memory, and how to use a tool called Ollama to download, run, and query a model locally. Once you can do this, you unlock the path to deeper topics in your library: understanding quantisation (how to shrink a model to fit), inference optimization (how to make it respond faster), and eventually edge and air-gapped inference for devices where the internet never reaches.

Imagine you've written a long story and you want to keep it in your pocket. You could carry the full manuscript, but it's heavy. Or you could condense it to a few pages of summary, losing a little detail, but now it fits easily. A large language model (LLM) is like that manuscript—a huge file on a server somewhere, containing learned patterns from terabytes of text. When you use ChatGPT, you're not carrying the file; you're sending a request to a computer that has it, and it sends back an answer. Local deployment means copying that file onto your own machine and running the model there. But models are enormous—a 7-billion-parameter model takes up several gigabytes of memory. Your computer might have 8, 16, or 32 GB of RAM. If the model doesn't fit, you have two choices: find a smaller model (like a pocket-sized summary) or compress the one you have (quantisation, which the library notes call the key enabler). Once it fits, you can run it as a local service: a program on your machine that listens for prompts and returns answers, just like a tiny version of ChatGPT that lives inside your computer.

Think of ordering food from a restaurant versus cooking at home. Ordering from a restaurant (calling an API) is convenient: you pick up the phone, tell them your dish, they cook it in their kitchen with their equipment, and send it to you. You don't worry about ingredients, stove, or chef skill—but you pay per meal, you can't change the recipe, and if the restaurant goes offline, you dine in the dark. Cooking at home (local deployment) means buying the recipe book (the model weights), having the right pots and pans (enough RAM and a GPU), and doing the cooking yourself. The first time is fiddly, but after you've done it, you can cook any time, you customise the recipe, you eat offline, and the marginal cost of each meal is just electricity. The analogy breaks down here: a restaurant can cook dishes you've never seen (the API model is often smarter), and a home kitchen is limited by its utensils (your hardware's memory and speed). So local deployment is a tradeoff: you gain privacy (your data never leaves your home), cost predictability (no per-token billing), and offline use, but you lose the capability ceiling and you take on the responsibility of maintaining your own 'kitchen'—installing software, ensuring enough memory, and troubleshooting when something breaks.

Local model deployment is the act of installing a trained machine learning model onto your own hardware (a laptop, desktop, or single-board computer) and running it there, so that you can send prompts and receive outputs entirely without internet access or a third-party API.

This concept sits at the point where machine learning meets practical, everyday software. In your library's structure, it belongs to Inference Optimization—the parent topic that covers quantisation, distillation, batching, and caching. You have notes on Quantization, which is the most immediate tool for making a model fit locally; Edge and Air-Gapped Inference, which extends local deployment to devices that must never touch the internet; and Latency and Speedups, which becomes relevant when you want your local model to feel fast. If you've used ChatGPT or any API, you already know the 'client' side; here you learn the 'server' side. The mathematics you need is just basic arithmetic—counting bytes and converting between megabytes and gigabytes—and the programming is mostly running a few commands and writing a tiny bit of Python to talk to the model.