Encoder-Decoder

Encoder-decoder is the architecture behind machine translation, text summarization, and image captioning. It's also the backbone of modern LLMs—the transformer encoder-decoder. Mastering this unlocks Transformers, Attention, and the ability to build systems that map any input sequence to any output sequence, regardless of length. It leads directly to further study of sequence-to-sequence models and attention mechanisms.

Think of the encoder-decoder as a two-stage process: first, read and understand; second, produce. The encoder reads the entire input and compresses it into a single, dense representation—a thought vector. This vector captures the meaning of the input, not its exact words. The decoder then takes this thought vector and generates the output step by step, word by word. The key insight is that the input and output lengths don't have to match. In translation, "How are you?" (3 words) becomes "Wie geht es dir?" (5 words). The encoder compresses the 3-word input into a thought vector, and the decoder expands it into a 5-word output. This separation of understanding from production is what makes the architecture so flexible. For example, in a chatbot, the encoder compresses the user's question, and the decoder generates a response. The encoder must understand the input completely before the decoder can start writing anything sensible.

Imagine a professional translator working at the UN. She listens to a speech in French (the input). She doesn't translate word-by-word as she hears it; instead, she forms a complete understanding of the meaning—the key points, the tone, the nuances. This understanding is her internal 'thought vector.' Only after the speaker has finished does she deliver the English translation, sentence by sentence, drawing on that full understanding. The encoder is the listening and comprehension phase. The decoder is the speaking phase. The thought vector is the complete mental picture. This analogy stops working, however, because a real translator retains far more detail than a fixed-length vector can hold. The encoder-decoder's thought vector is a compressed representation, so long inputs lose information. In practice, this 'bottleneck' is a real limitation—that's why attention mechanisms were invented. The translator doesn't forget the speech; she has access to all of it. The encoder-decoder without attention, by contrast, must remember everything in a single fixed-size vector, which is a hard constraint.

An encoder-decoder is an architecture that uses one neural network (the encoder) to compress an input sequence into a fixed-length representation, and a second neural network (the decoder) to generate an output sequence from that representation, allowing input and output lengths to differ arbitrarily.

You have notes on LSTM Networks and the LSTM Cell State. The encoder and decoder are often built from LSTM or GRU cells, which process sequences step-by-step. The encoder's final hidden state is the thought vector that seeds the decoder. This also connects to your notes on Layer Normalization, which stabilizes training in deeper encoder-decoder stacks. The key difference from a simple sequence model is the two-stage split: one network reads, the other writes.