Foundation Models

Foundation models are the reason you can use AI without building it from scratch. When you use ChatGPT, a translation app, or a photo search on your phone, you're almost certainly using a foundation model that someone else trained. Understanding this concept tells you: why one giant model can do hundreds of different jobs; why companies talk about 'adapting' or 'fine-tuning' instead of 'training from zero'; and why the same model can be both brilliant and biased. This page is the gateway to the rest of your learning path – it unlocks everything from prompt engineering to retrieval-augmented generation to AI agents. You will literally not understand any advanced AI topic without this foundation.

Before foundation models, you built software for one job at a time. A spam filter was trained on spam emails and can only filter spam. A face recognizer was trained on faces and can only recognize faces. Each one is a separate, expensive project. A foundation model flips this. You train one enormous model on a huge and varied chunk of the internet – text, images, code, audio – and it learns general patterns. Once trained, you can use that same model for thousands of different tasks: answering questions, writing emails, identifying objects in a photo, summarizing a document, translating a sentence. You don't train a new model for each task; you just 'prompt' the foundation model with a description of what you want. It's like the difference between a purpose-built factory that makes only toasters, and a skilled human worker who can make a toaster, a chair, or a birthday cake depending on the instructions. The foundation model is the skilled worker. You still need to tell it exactly what you want, but you don't have to build a new worker every time.

Think of a foundation model as a general doctor. You go to the same doctor for a fever, a broken arm, or a skin rash. The doctor didn't train separately for each disease – they went to medical school, learned the fundamentals of the human body, and now they can apply that knowledge to whatever you walk in with. A foundation model is the same: it 'went to school' on a huge body of text and images, and when you ask it a question or give it a new picture, it uses that general knowledge to respond. But the analogy breaks down in a crucial way: a doctor has a single mind that understands the world. A foundation model has no such mind. It's a giant pattern-matching engine. It has never 'understood' anything; it has simply learned to predict what the next word (or image pixel) should be, based on patterns in its training data. When you ask it a question, it's not recalling a fact – it's generating a string of text that statistically matches the pattern of a good answer. So a doctor can reason, but a foundation model can only imitate. And that imitation can be brilliantly useful, but it's also why it can confidently state a falsehood as if it were true – it's just following a pattern, not checking a fact… And just as a doctor's training shapes how they treat you, a foundation model's training data shapes everything it does – including its biases. If the training data is mostly one kind of person, the model will answer differently for people who look like that kind.

A foundation model is a large machine-learning model trained on a broad and diverse dataset, intended to be adapted to many downstream tasks via fine-tuning or prompting, rather than being trained for a specific task from scratch.

You have not learned anything yet, so this page is your starting point. But it connects forward to everything you will study. When you later learn about Large Language Models, you'll see that they are the text-based examples of foundation models. Concepts you'll meet later – by-pe encoding, context window, temperature – all interact with the foundation model idea.For example, your notes on Context Window (a neighbouring topic) explain that a model can only consider the tokens inside its window for any given call; a foundation model with a small context window can't handle a long document. Your notes on LLM Nondeterminism say that a foundation model can give different answers to the same prompt because it samples from a probability distribution – that's part of what it means to 'adapt' to a task. This page will not teach those, but it will set you up so that when you see 'fine-tuning', 'pretraining', or 'prompting', you'll know what they mean.