Continued Pre-Training

Continued pre-training is the step that gives a base model the vocabulary and knowledge of a specialized field before you fine-tune it for a task. Master this and you can adapt ChatGPT-style models to medicine, law, finance, or your company's internal jargon. This unlocks: knowing when to use continued pre-training vs. plain fine-tuning, avoiding catastrophic forgetting (your model forgetting general skills), designing the right data mix, and eventually combining it with instruction tuning and LoRA to build practical AI systems that actually understand your domain.

Imagine you are teaching a new employee who has a university degree in general studies but knows nothing about your specific industry. They can read and write fluently, but when you say "the counterparty defaulted on the swap," they understand each word separately but not what the phrase means together. You have two options: send them to get another degree (that's pretraining a model from scratch — far too expensive), or just give them a few months of reading your company's documents, contracts, and industry reports. After reading those, they still need to learn the specific tasks you want them to do, but now they can at least understand what they're reading. Continued pre-training is exactly that second step: taking a model that was trained on the whole internet, and letting it read a large amount of domain-specific text so it learns the words, phrases, and facts of that field, before you ever teach it a specific task. The model is not learning how to answer questions or follow instructions — it is learning the raw material: the language of the domain.

Think of a chef who trained in French cuisine. She is brilliant, but you are opening a regional Indian restaurant, and she has never cooked with mustard oil, or used asafoetida, or seen a recipe for pakhala. You could send her to a five-year culinary school for that cuisine — but that is slow and expensive (like training a model from scratch). Instead, for two weeks, she just reads cookbooks, watches grandmothers cook, and tastes dishes. She is not learning to run a kitchen yet; she is just absorbing the flavors, ingredients, and techniques. That is continued pre-training: you are not teaching the model to answer questions or classify anything; you are giving it the raw ingredients — the vocabulary and the factual background of your domain. After those two weeks, she still needs to learn how to run your particular kitchen (that's fine-tuning), but she now understands what the recipes say. The analogy breaks, however, in one important way: the model is not a human with a body or taste buds. It only reads text, so it cannot taste or smell anything; it can only learn the patterns of words and facts that appear in the text. If the important knowledge is not in the documents, the model will never gain it.

Continued pre-training is the process of extending a base model's training on a large, domain-specific corpus of text, before any task-specific fine-tuning, so that the model learns the vocabulary, style, and knowledge of that domain, while leaving its general conversational abilities intact as much as possible.

You already have notes on Large Language Models and Pretraining. This concept sits between full pretraining (training from scratch on trillions of tokens — prohibitively expensive) and fine-tuning (training on a smaller, task-specific dataset). It is also closely related to your notes on Catastrophic Forgetting and Domain Adaptation, because continued pre-training is a kind of domain adaptation, and the risk that it will forget the general knowledge is the same risk you have noted for fine-tuning. Finally, on this page you will see words like 'loss', 'weights', and 'gradient' — all of these were introduced in your notes on exactly those topics, so you can rely on that here.