Sliding Window Sampling

This is how every large language model — ChatGPT, Gemini, Llama — is taught to read and write. When a model predicts the next word in a sentence, it learned that skill by training on millions of sliding windows. Understanding this concept unlocks: Transformer training → Context Window (how much the model remembers) → Temperature and Sampling (how it chooses words) → and eventually fine-tuning models for your own tasks. If you can build a sliding window, you can build your own tiny language model and watch it learn to speak.

Imagine you are reading a book to learn a story. You don't read the whole book at once; you read one paragraph at a time, moving your eyes forward. Each paragraph gives you a little bit of story, and you connect it to the next. A sliding window is the same idea for a computer: you give it a short chunk of text (say, three words) and ask it to guess the next word. Then you shift the chunk one word forward and repeat. The 'window' is the chunk of text the model sees at any moment. 'Sliding' means you move it along the text, like a magnifying glass you drag across a page. The 'stride' is how many words you move the window each step. If you move it one word at a time, you get a huge number of overlapping chunks — that's a lot of teaching examples. If you move it three words at a time, you get fewer chunks, but they don't overlap as much. The window size is how many words the model sees at once — that defines how much context it has to predict the next word. A window of three words is a small scope; a window of 1024 tokens is a big one (tokens are like words but not exactly — you'll learn that later). This process is the core of how a model learns language: it sees the pattern over and over, and adjusts its internal numbers (weights) to get better at prediction.

You are a chef learning to cook by watching a master chef. The master chef cooks a long dish — say, a 20-minute recipe. You can't watch the whole thing at once and remember every detail. Instead, you watch a short clip (like 5 seconds) of the chef's hands, then you try to guess what they do next. Then you move the clip forward a second and watch the next 5 seconds, and so on. The 'clip' is the sliding window, the '5 seconds' is the window size, and the '1 second step' is the stride. If you use a small stride (1 second), you get many overlapping clips — you see the same action many times, which helps you learn it well. If you use a large stride (5 seconds), you get fewer clips, but you cover more of the recipe per step. The window size determines how much context you see — a 5-second clip shows you only the immediate motion, but a 20-second clip shows you the whole technique. The chef analogy breaks down when you realize the model doesn't just watch — it predicts the very next second. Also, the model doesn't 'remember' past clips; each clip is independent. And the model doesn't improve by watching — it improves by adjusting its internal numbers based on whether its guess was right or wrong. So the window is not a memory; it's just the current view.

Sliding window sampling is the technique of generating many small, overlapping training examples from a long text by moving a fixed-size window across the corpus, with a given stride (step size) controlling how much the windows overlap.

You have not studied any other topics yet. This is your first building block. Everything else in large language models — the context window, tokenization, attention — builds on the idea that the model is trained on these small, fixed-length snippets. Context Window is the concept that, at inference (when you ask the model something), the window is the maximum number of tokens the model can see at once. Sliding window sampling is the training-time version: you create many windows from a long text to teach the model. Later, when you learn about tokenization (how text is split into tokens), you'll see that the sliding window operates on token sequences, not raw characters. This foundational step is what makes next-token prediction possible.