Autoencoders

Autoencoders give you a way to compress data into a smaller essence and then rebuild it. That turns out to be useful in many real-world tasks: removing noise from photos, spotting fraud or machine faults, and creating the embeddings that RAG systems use to search documents. Understanding autoencoders is a direct step toward studying embeddings, anomaly detection, and representation learning. Later, when you meet LLMs and their internal representations, you'll recognise the same trick: learn a compact, meaningful form of the data rather than memorising the raw data itself.

An autoencoder is a kind of neural network that is trained to copy its input to its output, but the catch is that in the middle it is forced to make the information much smaller. So the network has to find a 'short description' of the input that still contains enough to reproduce it. That short description is the compressed representation, and the whole process is learning useful structure.

Think of a person who takes detailed meeting notes. If the person writes every word down, the notes are long and almost a copy. But if the person writes a short summary of only the key points, the summary loses some details, but still captures the main ideas. An autoencoder is like that: the input is the full meeting, the compressed middle is the summary, and the output is an attempt to rebuild the meeting from the summary. Because the summary is small, the network cannot simply remember every word—it has to learn what matters most.

In practice, the network has two parts: an encoder that compresses the input down to the small middle, and a decoder that expands the middle back to the original size. Training is done by giving the network many examples, calculating how different the output is from the input, and adjusting the weights to reduce that difference. Once trained, the middle part can be used as a compact fingerprint of the input, and the output can be used to regenerate something close to the original.

Imagine a photographer who has to send photos to a friend over a very slow connection—they have to compress each photo into a tiny file. The photographer learns that certain patterns (faces, trees, sky) can be stored with fewer bits. For a photo of a cat on a sofa, the photographer might encode 'cat, sofa, window-light' into a short code, and then the friend can generate a picture from that code. If the code is too small (too short), they lose important details; if it's large enough, the reconstruction looks almost like the original.

That is exactly what an autoencoder does: the encoder is the photographer compressing, the decoder is the friend reconstructing, and the size of the code is the bottleneck. The network learns what patterns in the images are worth keeping.

Where the analogy stops: the photographer manually decides what to encode, but the autoencoder discovers the features itself from data. Also, the reconstruction is a best-effort—the output is not pixel-for-pixel identical unless the bottleneck is big enough to hold all the information, which would defeat the purpose and is exactly what we avoid.

An autoencoder is a type of neural network that is trained to output a close reconstruction of its input, but with a middle layer that is forced to be smaller than the input, so the network has to learn a compressed representation of the data.

You have not studied any neural networks yet, so this page does not assume you know about weights, layers, or training. It only needs the idea of a function: something that takes a number (or a list of numbers) and produces another number (or list). The autoencoder is a function that maps a list of numbers (the input) to another list (the output), and we train it to make the output as close as possible to the input. The twist is that in the middle of the function there is a step where the list becomes much smaller—that is the bottleneck. Later, if you study embeddings or RAG, you will see that the same idea of a compact representation is used there.