Multi-Head Attention

Multi-head attention is the engine inside every modern large language model — ChatGPT, Claude, Gemini, and the open models like Llama all run on it. Without this page, the phrases 'attention is all you need' and 'multi-head self-attention' are just noise. With it, you can read a model architecture diagram and see what each component does, understand why models can look at an entire sentence at once, and grasp why scaling up the number of heads helps. This concept unlocks the deeper study of Transformers, causal masking, positional encodings, and the GPT architecture — all the machinery that makes generative AI work. In practical terms, after this page you'll be able to explain to a colleague why a model can correctly link 'she' to 'Maria' in a long document, and why throwing more heads at a problem isn't always better.

Imagine you are a detective trying to solve a mystery in a room full of people. You have one spotlight. You point it at the person who just spoke, then at the person they might be referring to, then at the person who looks nervous. You can only look at one thing at a time, and you have to decide where to point the light. That is single-head attention. Now imagine you have a team of detectives, each with their own spotlight. One detective always focuses on who is speaking, another on who is being talked about, another on where people are standing, another on the timeline of events. They all search simultaneously, in parallel, and then they pool their findings. That is multi-head attention. Each detective is a 'head', and each one specialises in a different kind of relationship. The key insight is that by running several attention operations in parallel, the model can capture different kinds of connections at the same time — syntax, position, meaning — without them interfering. In the original Transformer paper, they used 8 heads. Each head looks at a 'projected subspace' — a fancy way of saying that each head gets a slightly different filtered version of the same information, so it can focus on its specialty. The outputs from all heads are then concatenated — joined together into one long list — and passed to the next layer. The total computation stays roughly the same as a single larger attention operation, because each head works on a smaller dimension.

Think of a large tech company's board meeting. The CEO (the query) wants to know the status of every project. The project leads (the keys) present their updates. The actual progress reports (the values) are the details. In single-head attention, the CEO listens to each project lead, scores how relevant each update is, and then takes a weighted average of the reports — focusing mostly on the most relevant one. That works, but the CEO can only pay attention to one aspect of the reports at a time. What if the CEO could clone themselves? Clone A focuses exclusively on budget numbers. Clone B focuses on timeline delays. Clone C focuses on team morale. Each clone has a different 'lens' — they look at the same reports but filter them differently. After the meeting, all clones merge their notes into a single summary. That is multi-head attention. The clones are the 'heads', and each has its own projection of the queries, keys, and values — its own filter. The important detail is that the clones don't interfere with each other; they work in parallel, and their combined report is richer than any single clone could produce. But this analogy breaks down in one important way: the heads are not manually assigned specialties. The model learns what each head should focus on during training. Sometimes heads specialise in syntax, sometimes in coreference (like linking 'she' to 'Maria'), sometimes in position. And we don't always control what they learn — some heads end up redundant and can be pruned without hurting performance.

Multi-head attention is a mechanism that runs several attention operations in parallel, each on a different projected subspace of the inputs, then concatenates the results, so that different heads can learn to specialise in different kinds of relationships.

Your library notes under Transformers say that self-attention computes a weighted mixture of all positions, with weights derived from content. Multi-head attention is a direct extension of that: instead of one weighted mixture, you compute several, each with its own projected queries, keys, and values, and then combine them. You've also noted that causal masking is the single change that turns a bidirectional encoder into a generative decoder — multi-head attention is what gets masked in that case. And you have notes on positional encodings; the heads operate on the positions that those encodings inject. This concept builds on your notes about self-attention from scratch, and it leads directly to understanding the GPT architecture, where multi-head attention is stacked in layers.