Self-Attention from Scratch

Self-attention is the single most important idea in modern AI. It is the engine inside every large language model (LLM) — ChatGPT, Claude, Gemini, Llama — and it is also what makes retrieval-augmented generation (RAG) work, what lets a model on your phone complete your sentences, and what agentic systems use to decide which tool to call next. Understanding it from scratch means you will never again be lost when someone mentions 'attention', 'transformer', or 'context window'. It unlocks the next topics in your library: Causal Attention Masking (how a model generates one word at a time), Multi-Head Attention (how models track several relations at once), and ultimately the GPT Architecture. Without self-attention, none of these make sense.

Imagine you are reading a sentence and you come to the word 'it'. To understand what 'it' refers to, you glance back at earlier words — 'the cat sat on the mat, and it looked sleepy'. The word 'it' pays attention to 'cat' more than to 'mat', because 'cat' is the better candidate for the pronoun. You are scoring every earlier word by relevance and using those scores to blend their meanings into your understanding of 'it'. Self-attention does exactly this, but in arithmetic and for every word at the same time. Instead of a human eye, it uses three lists of numbers per word: a query (what this word is looking for), a key (what each word offers), and a value (the content that gets passed along). The score between any two words is how much their query and key agree. High agreement means the word pulls a large share of that word's value into its own updated meaning. The result is that every word becomes a weighted mixture of all other words, where the weights are learned—they are not hand-coded but tuned by training on huge amounts of text. This is why self-attention is so powerful: it lets a model directly connect distant words without passing through a long chain of intermediate steps. A sentence like 'The trophy did not fit in the suitcase because it was too big' is easy for self-attention because 'it' can directly attend to 'trophy' or 'suitcase' based on learned relevance.

Think of a team meeting where every member must summarise what they heard, but they are allowed to borrow ideas from others. Each member has a notepad with three columns: what they want to know (query), what they offer (key), and what they actually say (value). To decide how much to borrow from any colleague, they look at their own query and the colleague's key—if the key answers the query well, they copy a large portion of that colleague's value into their own summary. Each person does this for everyone else, and then their final summary is the weighted sum of everyone's values, with weights based on those query-key agreements. Crucially, this happens for all members simultaneously, not one at a time, so it is fast on parallel hardware. The analogy breaks in a few ways. First, in the meeting, people can choose to ignore someone; but in self-attention, every word always takes a bit of every other word—some bits are just tiny. Second, the query and key are not words themselves but learned numbers, and their agreement is measured by a dot product, not by human judgment. Third, the weights are not fixed per person; they change depending on every word, so the same word 'bank' has a different query for 'river' than for 'money'.

Self-attention is a mechanism that, for every position in a sequence, computes a weighted mixture of all positions (including itself), where the weights are derived from the compatibility between a learned query at that position and a learned key at each other position, and the mixture is formed from learned value vectors.

You have not studied any prior topics in your library, so this is your first deep insight into modern AI. You will notice that this page stands alone. Once you have it, you will see it everywhere: the next topic in your library is Causal Attention Masking, which adds a rule to self-attention so a model can only look at past words—this turns it into a generator. Then Multi-Head Attention runs several self-attention operations in parallel so different 'views' can emerge. And the GPT Architecture stacks these blocks. But you do not need any of those yet. The mathematics you need here is just arithmetic: multiplication, addition, and division. You will also meet the dot product—a simple way to combine two lists of numbers into a single score—taught from scratch on this page.