Edge and Air-Gapped Inference

You are learning this because it unlocks the ability to deploy AI where the internet does not reach. Concretely, after this, you will be able to understand how models run on your phone, on a factory floor computer that is physically cut off from the outside world, or in a hospital where patient data legally cannot leave the building. This leads directly to your next topics: Local Model Deployment (running models on your own hardware) and Quantization (shrinking models to fit). Without edge and air-gapped inference, you cannot use an LLM offline, cannot run a voice assistant on a smart speaker without cloud calls, and cannot build a system that survives a network outage.

Think of a model like a well-trained chef. The chef knows many recipes (the knowledge is in the model's weights). To serve a dish, the chef needs ingredients (the input) and a kitchen (the hardware). Now imagine two situations. First: the chef lives in a remote village with no market deliveries — they must cook only with what they can grow locally, and they cannot call a mentor for advice. That's edge inference: the model lives on the device (phone, sensor, car) and works with only that device's memory and compute. Second: the chef works in a kitchen inside a high-security government building where no one can enter or leave, and no phone calls are allowed. That's air-gapped inference: the model and the data stay inside a physically isolated network, never touching the internet. In both cases, the chef cannot fetch missing ingredients from a distant warehouse (no cloud API calls). They must work with what they have. The constraints are not optional — they are the whole point. For edge, it is about speed, privacy, and reliability (no network, no latency). For air-gapped, it is about security and legal compliance (data cannot leave). The key ideas from your library — quantisation (shrinking the model) and distillation (teaching a small model to mimic a big one) — are exactly the tools that make these situations possible, because memory is tight and updates are hard.

Imagine you are a librarian in a small village library. The library has a tiny building (device memory) and no phone line to the big city library (no internet). A villager asks for a recipe for a complex biryani. You have two options: you can walk to the big city every time (that would be a cloud API call — slow, and maybe not allowed if the village is in a locked compound). Or you can keep a small, well-chosen set of books inside your tiny library. That small set is your quantised model — it covers the most common requests, but it might not have every exotic recipe. If someone asks for a recipe not in your small collection, you cannot fetch it; you have to improvise or say no. In an air-gapped setting, the library is inside a military base; no one can bring in new books without a security check, and updates are done by physically carrying a USB stick through a guarded gate. That is the update path: slow, deliberate, and planned in advance. The analogy breaks down where a model is not a library but a compressed memory — the small collection is not a subset of the big city's books but a paraphrased summary, and some knowledge is inevitably lost or distorted, which is why quantisation has a quality cost. Also, unlike a librarian who can reason, the model is just arithmetic; it cannot improvise a truly novel recipe.

Edge and air-gapped inference is the practice of running a machine learning model entirely on a local device or inside an isolated network with no internet connection, which forces the model to be small enough to fit in local memory and designed so that its weights can be updated only through a carefully planned manual or scheduled process.

You have not yet mastered any prerequisite concepts, but your library notes on Inference Optimization mention that this topic belongs there. It connects forward to Local Model Deployment (running models on your own hardware) — edge is a special case where that hardware is a small device. It also relies heavily on Quantization (shrinking weights to save memory) and Knowledge Distillation (training a small student to imitate a large teacher) — without those compression tricks, almost no modern model would fit on an edge device. Finally, it touches Latency and Speedups, because edge models often trade a little quality for much lower response time. In the book The Craft of Post-Training, this is covered under the post-training stack, because quantisation and distillation are typically applied after the model is trained.