In words
What it is, why it matters, and what it is like.
Why am I learning this?
Cross-attention is the mechanism that lets a transformer read from one source while generating another. Imagine you are writing a description for a photo you see on your screen; you look at the image (the source) to decide what words to write next (the output). Mastering this concept explains how models perform tasks like turning an image into text, translating languages by looking back at the original sentence, or pulling specific facts from a large document to answer a question. It is the core technology that allows artificial intelligence to combine information from different types of data, such as matching words with pictures.
The idea, in plain terms
Imagine you are a professional translator sitting at your desk. On your left is a document written in French. On your right is a blank screen where you will type the English translation word by word. You do not just look at the last English word you typed; instead, for every new word you write, you glance back at the entire French document to find the best match. That French text acts as your reference or context.
Cross-attention is the mathematical process that enables this behavior in a computer model. It allows the model to pull specific, relevant information from a separate source each time it generates a new piece of output. It is called 'cross' because the request for information (the query) comes from one sequence—the part of the translation being built right now—while the information available to search through (the keys and values) comes from another sequence—the original French text.
Consider the specific numbers in the calculation. The model first measures how relevant each word in the source is by multiplying the request by the available data. Because these multiplication results can become very large numbers that are hard to work with mathematically, the system divides each result by the square root of the dimension size (a number representing the length of the vector representation). This step is called scaling and keeps the numbers stable. Next, the system uses a function called softmax to convert these scaled scores into probabilities. Softmax works by raising each score to the power of e (Euler's number), then dividing each result by the sum of all such results for that step. The output is a set of percentages that add up to 100%, showing exactly how much attention to pay to each part of the source.
Because these steps use information from two different sequences—asking about one thing while looking at another—there is a direct, content-based connection between them.
An analogy
Think of a chef in a busy kitchen following a long, detailed recipe card placed on the counter. The recipe card is the source of truth, and the dish being plated is the output. Each time the chef needs to add an ingredient or describe a step, they glance at the recipe to see what comes next. The chef does not re-read the entire card from the top every time; instead, they focus their eyes on the specific line relevant to their current action. In this analogy, the chef’s immediate task—‘what should I do now?’—is the query. The lines on the recipe card are the keys and values: the headings or bolded words act as keys that tell the chef what topic each line is about, while the text below them are the values containing the actual instructions. The chef mentally matches their current task to the most relevant line in the recipe and follows those specific instructions.
Where this analogy breaks down: a human chef has common sense and understands that ‘simmer for 10 minutes’ implies time passing, whereas a model has no such innate understanding and must learn these associations purely through training. Also, while a human reads lines sequentially, a model can weigh the relevance of every line in the source simultaneously.
Definition
Cross-attention is a process where a model generates a sequence by repeatedly looking at a separate source sequence for context. It calculates relevance by measuring the match between the current step of generation (query) and all parts of the source (keys), converting those matches into weights (values) that determine how much information to borrow from the source.
Where this sits
This sits next to Self-Attention, which is a process where a model looks at every part of a single sequence to understand its own internal relationships. In self-attention, the queries, keys, and values all come from the same text. In cross-attention, the queries come from the generating side (often called the decoder), while the keys and values come from the reference side (often called the encoder) or a different type of data (a modality, such as an image versus text). It is also closely related to Multi-Head Attention, which is simply the technique of running several cross-attention processes in parallel to capture different kinds of relationships at once.
