Causal Attention Masking

Causal attention masking is the single change that turns a bidirectional encoder into a generative decoder. Without it, a transformer would 'see' future tokens during training and be useless for generation. This concept unlocks the study of GPT architecture, autoregressive training, and eventually the whole of large language models. If you want to understand how ChatGPT writes text, this is your first door.

Imagine you're writing an essay, word by word. When you write the second word, you don't know what the third word will be yet. You can only use the words you've already written. That's exactly what causal attention masking does: it forces the model to predict each next word using only the previous words. It's like having a strict editor who hides the rest of the page so you can't cheat by peeking. In the transformer, each position 'attends' to all other positions. Causal masking says: position 3 can attend to positions 1, 2, and 3, but not 4, 5, etc. This is how the model learns to generate text one token at a time, and why it can be trained on massive text corpora in parallel.

Think of a class of students copying a sentence from the board, one word at a time, but with a cover that hides all words after the one they're currently copying. The teacher shows the first word, they copy it. Then the cover moves to reveal the second word, they copy that, and so on. The students can never see a future word. Now, this is training: each student is being tested on their ability to predict the next word from the ones they've seen. The mask is the cover. In a transformer, the cover is a matrix that zeros out any attention score for future positions. After the mask, the softmax makes those future influences exactly zero. The analogy breaks down when we consider parallel processing: in the classroom, students work one word at a time, but with the mask, the transformer can process all positions simultaneously because the mask prevents any information from flowing backward. So, training is fast and parallel, but the model still behaves as if it were sequential.

Causal attention masking is the operation of setting attention scores for future positions to negative infinity before applying softmax, so that each position in a sequence can only attend to itself and earlier positions, enabling the model to generate text autoregressively.

You have notes on Self-Attention from Scratch, Multi-Head Attention, and the Transformer architecture. In self-attention, every position attends to every other. Causal masking is a modification to that: you keep the same machinery but zero out the future. This is what makes the difference between an encoder (like BERT, which reads the whole sentence) and a decoder (like GPT, which generates one token at a time). Your note from "Build a Large Language Model (From Scratch)" says exactly this: 'Causal masking is the single architectural change that turns bidirectional self-attention into a generative language model.' That's the key insight to hold onto.