Positional Encoding

You are studying Transformers, the architecture behind ChatGPT and other modern large language models. These systems use a mechanism called self-attention to understand text. This mechanism allows every word in a sentence to look at every other word to understand context. However, this method treats words as an unordered collection, like a pile of scattered letters where 'cat' and 'act' are indistinguishable. Without a way to track order, the model cannot distinguish between 'dog bites man' and 'man bites dog'. Positional encoding is the solution that restores this sense of order. It is the foundational step that makes it possible for these models to read, write, and understand language structure.

Imagine a meeting where everyone is speaking at once. Each person can hear every other voice, but without knowing who spoke first, second, or last, the conversation is just noise. In a Transformer, the self-attention mechanism lets each word 'hear' every other word to gather context, but it has no natural sense of time or sequence. If the model sees the words 'river', 'bank', and 'loan', it knows their meanings individually but does not know that 'bank' follows 'river' and precedes 'loan'.

Positional encoding solves this by tagging every word with a specific marker of its position in the sequence as it enters the system. Think of it like giving each participant at the meeting a badge that displays their turn number. The model then uses two pieces of information for each word: what the word means (from the previous learning stage) and where it sits (from the positional encoding). By combining these, the model can distinguish that the sequence 'river bank' has a different meaning than 'bank river'.

For example, if you have the sentence 'The cat sat on the mat', the first word 'The' might be tagged with position 1, the second word 'cat' with position 2, and so on. The encoding ensures that the model knows 'cat' is the subject acting in the first half of the sentence, not the object at the end.

Imagine you are arranging a photograph album. The photos themselves represent the content of the words, but the order in which they appear tells the story. If you scatter the photos randomly on the floor, the narrative is lost. A positional encoding is like numbering each photo on the back before you put them in the album. These numbers tell you exactly where each photo belongs.

However, simple integer numbers (1, 2, 3) have a limitation: they do not naturally convey how close two items are relative to the total size of the set. For instance, the distance between photo 5 and photo 6 is physically one step, but if the album has 100 photos, that step is a small fraction (1/94) of the remaining space. If the album only has 10 photos, that same step is a much larger proportion of the journey. To handle this better, advanced systems use values that shift smoothly in a wave-like pattern instead of simple integers. These shifting patterns create structures where positions close together have similar values, and positions far apart have different ones. This allows the model to easily calculate relative distances—for example, knowing that position 5 is very close to position 6, regardless of whether the total sequence length is 10 or 100.

This analogy has one limitation: in a Transformer, the positional information is not just written on the back and kept separate; it is mathematically added to the word's meaning list, so the two are fused together from the start.

A positional encoding is a set of numbers added to each word's representation (embedding) to indicate its specific location in the sequence, allowing the model to distinguish order despite processing all words simultaneously.

This concept builds directly on your understanding of embeddings, which are lists of numbers that capture a word's meaning. While embeddings provide the 'what', positional encodings provide the 'where'. It is also closely tied to self-attention, the mechanism where each word evaluates its relationship with every other word; without positional encodings, self-attention would be blind to sequence structure.