Positional Encoding

In your study of Transformers (the architecture behind ChatGPT and every modern large language model), you will soon meet self-attention—the mechanism by which each word in a sentence looks at every other word. But self-attention, as it is first defined, has no sense of order: it treats a sentence as a bag of words, which is fatal. Positional encoding is the invention that restores order, and it is the reason Transformers can understand language at all. This concept directly unlocks Causal Attention Masking, Multi-Head Attention, and RoPE Rotary Embeddings, which you will need for GPT Architecture and eventually for building your own small language model. Without positional encoding, no Transformer-based AI would work—this is the foundation stone.

Picture a meeting where everyone is talking at once. Each person can hear every other, but without knowing who spoke first or last, the conversation is meaningless. Attention in a Transformer works similarly: it lets each word look at every other word to gather context. But if the model can't tell that 'bank' came after 'river' but before 'loan', it can't grasp the meaning. Positional encoding is the system that tags every word with its position number as it enters the model. It is like giving each participant a badge with their order in the queue. The model then uses this badge to consider both what a word says and where it appears.

Imagine you are arranging a photograph album. The photos themselves (the word contents) matter, but so does the order—the sequence tells a story. If you scatter the photos randomly, the story is lost. A positional encoding is like numbering each photo on the back before you put them in the album. The number tells you where each photo belongs, so even if you shuffle them on the table, you can restore the correct order. But a simple number (1, 2, 3...) has a weakness: it doesn't tell you how far apart two photos are relative to their total count. If you have 100 photos, photo 5 and photo 6 are close; if you have 10 photos, they are a significant gap. The sinusoidal encodings (using sine and cosine waves) solve this by representing positions as coordinates on a wave, where nearby positions have similar patterns and faraway ones differ smoothly—so the model can infer relative distances. However, this analogy breaks down because the model doesn't just use the number; it adds the encoding to the word's meaning, so positional information and content are fused together, not kept separate as in an album.

A positional encoding is a vector (a list of numbers) added to each word's embedding to inform the attention mechanism of that word's position in the sequence, so that order matters even though attention itself is unaware of it.

You have likely learned about embeddings, where each word is converted into a list of numbers that capture its meaning. Positional encoding is an extra list of numbers, added to that embedding, to capture position. In the Transformer architecture, self-attention computes scores between every pair of positions—if you haven't met self-attention yet, it is coming next. Positional encoding is the prerequisite that makes self-attention possible for sequences. Your library notes mention that 'attention alone has no notion of sequence order at all'—this is the key problem. Also, you may have seen causal attention masking (a later concept) which prevents words from looking ahead; positional encoding is what allows them to know their place even when looking back.