LSTM Networks

You're about to meet the architecture that powered speech recognition, machine translation, and stock forecasting for a decade — and still quietly runs inside many of today's systems. Learning LSTMs gives you the mental model for how neural networks handle sequences: music, sensor readings, words in a sentence. It also sets you up to understand what came next. When you study transformers — the architecture behind ChatGPT — half the battle is understanding why they solved a problem LSTMs had: carrying information across long stretches without forgetting or losing the signal. And if you ever work on a device that must respond instantly, like a phone that listens for 'Hey Google', you'll find LSTMs still there, because they don't need to see the whole sentence before they start. This page teaches you the LSTM from the ground up — the four gates, the memory cell, and how they work together — so that later, when you meet attention, you'll recognize it as a clever twist on the same problem.

Picture a factory conveyor belt that carries a heavy box from one end of a long hall to the other. Along the way, workers stand at stations. At each station, a worker inspects the box, decides whether a part needs to be added, removed, or left alone, and then signals the belt to move on. The box is never dumped out and rebuilt — it's nudged, adjusted, and occasionally something is taken off or put on. That's the core idea of an LSTM cell: it maintains a piece of information — the 'cell state' — that travels through time, and at each timestep, it decides, based on the current input, what to forget, what to write, and what to output. The genius is that the cell state changes gradually, by adding or removing small amounts, rather than being overwritten. This is what lets the network remember something from ten steps ago: the signal doesn't have to travel through all the intermediate transformations; it just rides along the belt, gently modified, until it's needed. In a plain RNN, there's no belt — every step takes the whole state and recomputes it, so information from early steps gets diluted or multiplied by the same weight over and over, causing the gradient to either vanish or explode. LSTM's gates are small neural networks themselves, each with a weight and a bias, that produce a number between 0 and 1 — a 'how much to let through' factor. Forget gate: how much of the old cell state to keep. Input gate: how much of the new candidate to add. Output gate: how much of the cell state to reveal as the output. Put together, these gates let the LSTM learn which parts of the past to remember, which to forget, and what to say right now.

Imagine you're writing a long essay, and you keep a running notes sheet in front of you. At the end of each paragraph, you look at your notes and decide: 'The main argument from paragraph 2 is still relevant, keep it. The digression about the author's childhood — I can cross that out.' That's the forget gate. Then you read the new paragraph and think: 'This adds a supporting point, I should write it down.' That's the input gate, writing into memory. Finally, when you're about to start the next paragraph, you look at your notes and decide what to actually say in the essay — you don't read out your entire notes verbatim, you pick a summary phrase. That's the output gate. The LSTM is doing exactly this, but with numbers. The cell state is your notes sheet, a list of numbers that represent 'what I know'. The input at each time step is a new paragraph. The forget gate is a set of weights that decide, for each number in the cell state, how much to keep (a factor between 0 and 1). The input gate decides how much of the new information (a 'candidate' created from the input and previous output) to add to the cell state. The output gate decides what to 'say' — the hidden state — which is a filtered version of the cell state. The analogy breaks down where the real LSTM is more subtle: the gates are not separate conscious decisions; they're all computed simultaneously from the same input, and they are learned, not hand-crafted. Also, the cell state isn't a sheet of paper — it's a vector of numbers, and the gates apply element-wise, so each dimension of the cell state can be forgotten or updated independently. But the core intuition holds: a conveyor belt that carries a memory, with workers (gates) that add, remove, and read from it, and the key is that the belt allows the memory to flow with minimal interference, so distant information survives.

An LSTM (Long Short-Term Memory) network is a type of recurrent neural network that uses gated cells to control the flow of information through a memory state, allowing it to carry relevant information across long sequences without suffering from the vanishing or exploding gradient problem that plagues plain RNNs.

You haven't learned anything else yet, so this page is your starting point. The LSTM belongs to the family of recurrent neural networks (RNNs), which process sequences one element at a time and maintain a hidden state that carries information across time steps. The LSTM is a refinement of the plain RNN, which added a simple hidden state but couldn't remember long-range dependencies. The LSTM's innovation is the cell state — a separate memory track that is modified additively through gates. When you later study transformers, you'll see they replaced the recurrence with attention, which can look at any part of the sequence directly rather than incrementally, but the LSTM's lessons about memory and gating still inform how we think about attention.