GPT Architecture

This is the architecture behind ChatGPT, Claude, Gemini, and every other large language model you have used or will build. Understand GPT, and you understand how text generation works, why it sometimes hallucinates, what 'context length' means, and why a model cannot look ahead in a sentence. It is also the foundation for smaller models that run on phones, retrieval-augmented generation, and agentic systems. This concept unlocks the rest of the Transformers family: causal attention masking, multi-head attention, positional encoding, and RoPE.

Imagine you are reading a sentence word by word, and at each word you try to predict the next one. You cannot see the future — you only have the words so far and your own memory. GPT works in exactly that way. It is a machine that, given a sequence of words, produces a probability for every possible next word. It does this by passing the words through many identical 'blocks', each of which lets every word look at every earlier word (that is self-attention) and then applies a simple transformation to each word individually (the feed-forward part). The crucial rule is the causal mask: when the model computes the probability for the next word, it is never allowed to peek at words that come after. This is what makes it 'decoder-only' — it only generates, it does not read the whole input at once like a translator. The training is autoregressive: it tries to predict the next word, compares its prediction to the real next word, and adjusts its internal numbers (the weights) to make the prediction better. Over millions of examples, the model becomes excellent at predicting the next word, and that is all it ever does, even when it is answering your question or writing an essay.

Think of a very patient, single-minded storyteller who tells a story one word at a time. They have a rule: they can only use the words already said, never the words they are about to say. At each step, they consider all the words they know (the vocabulary), and give each a score indicating how likely it is to be the next word. They then pick one word (sometimes the most likely, sometimes a random one depending on 'temperature'), and that becomes the next word in the story, and then they repeat. The storyteller's 'brain' is made of many identical offices (the layers). In each office, two things happen: first, each word in the story so far is written on a card, and every card is compared to every earlier card to see how relevant it is — this is self-attention, where words exchange information. Second, each card is transformed by a small 'thought processor' (the feed-forward network) that does not look at other words, only at itself. The office then adds its own contribution to the original cards (residuals) and normalises the numbers (layer normalisation) to keep them stable. The storyteller repeats this office process over and over — say, 96 times for a large model — and at the end, they have a final set of numbers for each word, which they convert into probabilities for the next word. The analogy stops working in one important way: the storyteller is not a person with learned habits; every office is identical in structure, but each has its own set of numbers (weights) that are tweaked during training. Also, the storyteller does not actually understand words in a human sense — it only understands patterns in numbers.

GPT (Generative Pre-trained Transformer) is a decoder-only transformer that generates text one token at a time by attending to all previous tokens through stacked self-attention and feed-forward layers, with causal masking, residual connections, and normalisation, trained to predict the next word.

You have not studied any prerequisites yet, so this page starts from absolute basics. But you will soon learn about Causal Attention Masking, which is the single mechanism that makes this a decoder, and Multi-Head Attention, which lets the model attend to different relations. Later you will add Positional Encoding (since attention has no built-in sense of order) and RoPE embeddings. This page gives you the whole picture; the later topics will zoom into the parts.