Q-Learning

Q-learning is the foundation of reinforcement learning (RL) — the branch of AI where a program learns by trial and error, like a video game character that gets better the more it plays. Mastering Q-learning unlocks the entire RL family: Bellman equations, value iteration, deep Q-networks (DQN), and modern post-training methods (RLHF, DPO, GRPO) that fine-tune large language models. It also directly feeds into your notes on AlphaGo and MCTS — AlphaGo's value network learns by a similar 'how good is this state' idea. Even if you never build a robotics controller, understanding Q-learning tells you how an AI agent can learn a policy without being told the rules — the same mechanism behind recommendation engines that explore what you like, and agents that plan and call tools.

Imagine you are in a maze, and each square gives you a small reward (like +1 for reaching the exit, but you don't know the map). You can move up, down, left, right. At first you move randomly. But every time you move, you learn a little bit: 'Going right from this square led me closer to the exit, so that was good.' You build a mental table: for each square (state) and each direction (action), you store a score — how good that action is from that square. The more you play, the more you update these scores based on the rewards you actually see. After many plays, you know the best action from every square. That table is the Q-table — 'Q' stands for quality. Q-learning is the algorithm that fills in that table correctly, even though at first you are just guessing. It works by a simple rule: every time you take an action and get a reward, you adjust the score for that (state, action) pair a little bit toward 'the reward I just got plus the best score I could get from the next square'. You keep doing this until the scores stop changing. No one tells you the map — you learn it by trying.

Think of a delivery person in a new city who must learn the best route from any street corner to the depot. They start by wandering — sometimes turning left, sometimes right, because they don't know which way is faster. Each time they arrive at a corner, they get a 'reward': maybe +10 for reaching the depot, 0 for other corners, -1 if they hit a dead end. They keep a small notebook: for each corner (state) and each possible direction (action), they write a number — 'how good is it to go this way from this corner?' Initially all numbers are 0. When they take a step from a corner, they observe the immediate reward and then look at the notebook for the best number at the new corner. They update the number at the old corner toward 'reward + best at new corner'. For example, if going right from corner X leads to corner Y which has a notebook value of 5, and the step gave 0 reward, they set the value of (X, right) a bit closer to 5. Over many trips, the numbers converge to the true 'distance to depot' for each direction. Where the analogy stops working: the delivery person is trying to find the shortest path, but Q-learning can handle much more — like a game where rewards are random, or where actions can have delayed effects. Also, in reality, the agent doesn't have a 'notebook' that grows forever; in complex problems the table becomes too big, which is where deep Q-networks come in. But the core idea — update scores based on observed rewards plus best future estimate — is exactly Q-learning.

Q-learning is an algorithm that learns the value of taking a specific action in a specific state (the Q-value) by repeatedly updating an estimate toward the observed immediate reward plus the best estimated future reward, without needing a model of the environment's dynamics.

You have notes on Reinforcement Learning as a parent concept: learning a policy through interaction. Q-learning is one of the core algorithms under that. It builds directly on your notes about Bellman Equations — the update rule in Q-learning is exactly the Bellman equation applied to action-values. Also, it connects to your neighbouring notes on AlphaGo and MCTS: AlphaGo uses a value network (a neural network that approximates Q-like values) to guide Monte Carlo tree search. You have notes on 'exploration-exploitation tradeoff' — Q-learning is a classic example of handling that tradeoff. Finally, your library mentions RLHF and DPO for language models — those are modern variants of this same idea: the model learns to choose actions (words) that maximize reward (human preference).