In words
What it is, why it matters, and what it is like.
Why am I learning this?
Cross-attention is the mechanism that lets a transformer read from one source (like an image or a translated sentence) while generating another (like a caption or a translation). Master this and you unlock the architecture behind image captioning, text-to-image models, machine translation, and retrieval-augmented generation. It is also the foundation for understanding how AI systems combine information from different modalities, which is the next big frontier in applied machine learning.
The idea, in plain terms
Imagine you are a translator. You have a document in French in front of you, and you are producing an English sentence word by word. You don't just look at the last English word you wrote; you constantly refer back to the French sentence to decide what to write next. That French sentence is the 'source' or 'context'. Cross-attention is the arithmetic that lets a model do exactly this: pull relevant information from a separate source each time it produces a new piece of output. It is called 'cross' because the query comes from one sequence (the output so far) and the keys and values come from another sequence (the source), allowing a direct, content-based connection between the two.
An analogy
Think of a chef following a recipe. The recipe is the source, and the dish the chef is plating is the output. Each time the chef adds an ingredient or a garnish, they glance at the recipe to see what is next. The chef doesn't re-read the whole recipe each time; they focus on the specific line that tells them the next step. The chef's current question (what to do next) is the query. The recipe's lines are the keys and values: the keys tell the chef which line is about what, and the values are the actual instructions on those lines. The chef matches their question to the most relevant recipe lines, and then follows those instructions. Where this analogy breaks down: the chef has a fixed recipe, but in cross-attention the source can be of any length and the model can look at all parts of it simultaneously, not just one line at a time. Also, the model does not have a human's common sense; it has to learn what to 'glance at' through training.
Definition
Cross-attention is an attention mechanism where the queries are derived from one sequence, while the keys and values are derived from a different sequence, allowing the model to condition the generation of one sequence on the content of another.
Where this sits
This builds directly on your notes on Self-Attention from Scratch: the scoring, scaling, and softmax are identical to self-attention. The only difference is where the queries, keys, and values come from — in self-attention, all three come from the same sequence; in cross-attention, the queries come from the decoder's own output so far, while keys and values come from the encoder's representation of the source (in a translation model) or from a separate modality (like an image). It is the mechanism that makes Multi-Head Attention useful across sequences, and it is a key part of the GPT Architecture (though GPT uses only self-attention, cross-attention is the core of encoder-decoder transformers like T5).
