Self-Attention from Scratch

You know that when you read a difficult sentence, your brain doesn't process each word in isolation. It looks back at earlier words to clarify meaning, or looks ahead to understand context. Self-attention is the mathematical technique that gives artificial intelligence this same ability. It is the core engine behind ChatGPT, Claude, and other modern language models. Understanding it from scratch means you will never again feel lost when someone mentions 'transformers', 'context windows', or how AI handles long documents. It unlocks the next topics in your library: Causal Attention Masking (how a model writes text one word at a time without cheating), Multi-Head Attention (how models track several relationships at once), and ultimately the GPT Architecture. Without self-attention, none of these make sense.

Imagine you are reading this sentence: "The trophy did not fit in the suitcase because it was too big." Your brain immediately links the word "it" to "trophy," not "suitcase." How? By scoring how well each earlier word matches the current question. Self-attention does exactly this, but with numbers for every word at once.

Here is how the arithmetic works, step by step, using a tiny example with just three words: A, B, and C. We want to update the meaning of Word A.

1. The Setup: For each word, we have three lists of numbers (vectors). Think of them as simple tags:
- Query for A: [1, 0] (What A is looking for)
- Key for B: [2, 1] (What B offers)
- Value for B: [5, 5] (The actual content B provides)

2. Scoring Compatibility: To see how relevant B is to A, we calculate a "score." We multiply corresponding numbers in the Query and Key lists, then add the results together.
- Multiply first positions: 1 * 2 = 2
- Multiply second positions: 0 * 1 = 0
- Add them: 2 + 0 = 2. This is the score.

If we did this for every other word, we would get a list of scores. Let's say the scores are: A vs A = 3, A vs B = 2, A vs C = 1.

3. Weighting: We want to combine information from all words into A. But we care more about high-scoring words. We turn these scores into weights by dividing each score by the total of all scores (3+2+1=6). So A gets a weight of 3/6 (50%) from itself, 2/6 (~33%) from B, and 1/6 (~17%) from C.

4. Weighted Mixture: Now we blend the "Value" lists. A takes 50% of its own Value, plus 33% of B's Value, plus 17% of C's Value.
- If B's Value was [5, 5] and C's Value was [0, 10]:
- Contribution from B: 0.33 * [5, 5] ≈ [1.65, 1.65]
- Contribution from C: 0.17 * [0, 10] ≈ [0, 1.7]

This blend is the "weighted mixture." The word "it" in the original example has a high score for "trophy" and a low score for "suitcase," so its final meaning is dominated by "trophy"'s value. In arithmetic terms, high agreement (score) means that word pulls a large share of its value into the current understanding.

Think of a team meeting where every member must write a summary of what they heard, but they are allowed to borrow ideas from others. Each member has a notepad with three columns: what they want to know (Query), what they offer (Key), and what they actually say (Value). To decide how much to borrow from any colleague, they look at their own Query and the colleague's Key—if the Key answers the Query well, they copy a large portion of that colleague's Value into their own summary. Each person does this for everyone else simultaneously, so the final summary is a blend of everyone's input, weighted by how relevant each person was. The analogy breaks in one key way: in a meeting, people can choose to ignore someone completely; in self-attention, every word always takes a tiny bit of every other word—some bits are just vanishingly small. Additionally, the "agreement" between Query and Key is calculated by multiplying corresponding numbers and adding them up, not by human judgment.

Self-attention is a process that updates each item in a sequence by blending information from all items (including itself), where the blend ratio depends on how well each item's 'offer' matches the current item's 'search'.

This sits beside Causal Attention Masking, which adds a rule to this blending process so a model can only look at past words—this turns it into a generator. It also sits beside Multi-Head Attention, which runs several of these blending operations in parallel so different aspects of meaning can emerge simultaneously.