AlphaGo and MCTS

This is the story of how a machine beat the world champion at the most complex board game ever devised — and it is not brute force. AlphaGo combined two ideas: a fast way to guess good moves (a policy network) and a way to judge positions without playing to the end (a value network), wrapped around a search method called Monte Carlo tree search. Understanding this unlocks your path into Reinforcement Learning — the branch of AI where agents learn by trial and error — and later into modern techniques like RLHF (Reinforcement Learning from Human Feedback) that shape language models. You will see how self-play generates training data, how search plus learned evaluation beats either alone, and why test-time search (thinking longer before answering) pays off. These are the same principles behind AI agents that plan, call tools, and act in loops.

Imagine you are a chess player with only seconds per move. You cannot calculate every line to the end. Instead, you rely on instinct (which moves look promising) and a quick sense of who is winning. You might try one move, see the opponent's likely reply, then your reply, and so on for a few moves — but you prune the branches that look hopeless and spend more time on the promising ones. AlphaGo does exactly this, but at a scale no human can match. It plays out thousands of imagined games in its 'mind' in a fraction of a second, using its learned intuition to guide where to look and its learned judgment to evaluate the positions it reaches. The key insight: you do not need to see the entire tree of possibilities. You need a smart way to explore the most promising branches and a reliable way to judge when you have gone deep enough. That is Monte Carlo tree search — a way to balance exploring new moves against exploiting the ones that have already worked well in this position.

Think of a detective trying to solve a mystery. The detective has limited time and must decide which leads to chase. Each lead is a move in a game tree. The detective's intuition (policy network) says which leads look most promising based on experience. The detective's judgment (value network) says how close they are to solving the case — or in Go, how likely they are to win from a given board position. The detective does not follow every lead to the end. Instead, they follow a few promising ones, see what happens, and then decide where to focus next. They might even simulate what would happen if they took a wild guess and followed it for a while, then use that outcome to update their beliefs about that lead. That is Monte Carlo tree search: you repeatedly pick a move, play out a random or guided game to the end, and record whether you won. Over many repetitions, you build a statistical picture of which moves lead to victory. This works because of the law of large numbers — with enough samples, the average outcome converges to the true probability of winning. But in Go, the branching factor is so huge (about 200 legal moves at the start) that you cannot just sample randomly. You need guidance — that is where the policy network comes in. It tells you which moves are worth trying, so you don't waste samples on terrible moves. And the value network saves you from having to play each simulation to the very end — it estimates the outcome from the current position, which is much faster. Where the analogy breaks down: a detective can change their mind mid-investigation; MCTS in AlphaGo makes a decision at the root and then commits, though it re-searches each move. Also, the detective's intuition is static; AlphaGo's policy network gets better as it learns from self-play.

Monte Carlo tree search guided by learned policy and value networks — a search algorithm that builds a tree of possible moves, simulating games to estimate each move's winning probability, while using neural networks to prioritise which branches to explore and to evaluate positions without playing to the end.

Your library notes that AlphaGo is the narrative bridge from classical ML into modern AI — it is the clearest demonstration of test-time search paying off. It belongs to the Reinforcement Learning family: the agent (AlphaGo) interacts with an environment (the Go board), takes actions (placing stones), and receives a reward at the end (win or lose). It builds on the exploration-exploitation tradeoff — you must try new moves (explore) but also stick with moves that have worked (exploit). The Bellman equations, which relate the value of a state to its successors, are the mathematical foundation for the value network's estimate. Q-learning is the simpler cousin: it learns the value of taking an action in a state, but it becomes infeasible at scale — AlphaGo's networks compress that value function into a neural net. The policy network is a direct policy optimisation: it maps a board state directly to a probability distribution over moves.