Random Forests

Random forests are one of the most reliable tools for machine learning. They provide a strong starting point for almost any dataset organized in rows and columns, where each row is an example and each column is a piece of information about that example. This structure is called a tabular problem because the data looks like a spreadsheet. Random forests help you predict outcomes for new examples and tell you which pieces of information (features) are most important. They rarely require delicate tuning, meaning you can often get good results without spending hours adjusting settings. Learning this builds a solid foundation for understanding more complex systems. For instance, the idea of combining many simple predictions into one reliable result is a key ingredient in modern techniques that power advanced language models, which generate text based on patterns in massive amounts of writing.

A single decision tree can be unstable. A decision tree is a model that makes predictions by asking a series of yes-or-no questions about the data, following a path from the top to a final answer. If you change the training data even slightly, the structure of this tree can change completely, causing its predictions to jump around unpredictably. This instability happens because a single tree tries to memorize every detail of the specific data it sees.

Random forests fix this problem by building many decision trees and combining their answers. Instead of relying on one tree, the forest creates a large number of trees. Each tree is trained on a slightly different version of the data, created by picking examples randomly with replacement (meaning an example can be picked more than once, while others might not be picked at all). Additionally, when each tree decides which question to ask next, it only looks at a random subset of the available features. This double randomness ensures that the trees are different from one another.

For example, imagine you want to predict whether a customer will buy a product. One tree might focus heavily on age, another on past purchase history, and another on time spent on the site, because they saw different data or looked at different features. When you combine these diverse opinions, the errors of individual trees tend to cancel each other out, resulting in a more reliable and often more accurate prediction than any single tree could provide.

Think of a committee of experts making a decision, such as a group of doctors diagnosing a patient’s condition. If you asked just one doctor for a diagnosis, their answer might depend heavily on their specific training, the recent medical journals they read, or even the time of day. Their judgment could be influenced by rare cases they recently treated, leading to inconsistency.

However, if you assemble a diverse panel of doctors and ask each one to diagnose the patient independently, the group decision is usually more balanced and robust. This is how a random forest works. Each tree in the forest is like one doctor. The 'bootstrap sample' means that each doctor reviews a different set of past patient records (some records appear multiple times for one doctor but not at all for another). The 'random feature subset' means that when each doctor thinks about the diagnosis, they are only allowed to look at a random selection of the patient’s symptoms and test results, ignoring others temporarily. The final diagnosis is determined by the majority vote of all doctors.

Where this analogy has limits: If all the doctors were trained identically and looked at the exact same data, they would all make the same mistakes in the same way. Random forests avoid this by shuffling the data and limiting the information available to each tree, ensuring diversity. Also, while a single doctor can clearly explain their reasoning step-by-step, a committee’s collective decision is harder to trace back to a single clear cause, so random forests trade some interpretability for accuracy.

A random forest is a machine learning model that combines the predictions of many decision trees. Each tree is trained on a random sample of the data and uses only a random subset of features at each step. For problems where you predict a number (regression), the final result is the average of all trees' outputs; for problems where you predict a category (classification), the final result is the most common choice among all trees.

You have notes on Decision Trees, which are the individual building blocks here. A random forest is essentially many decision trees working together, but with crucial tweaks: each tree gets a different random sample of the data and considers only a random subset of features at each split. This diversity allows the average prediction to be more stable than any single tree. You can compare this to K-Nearest Neighbours, another method that predicts based on similar past examples, but while K-Nearest Neighbours looks at local similarities for every new question, a random forest combines many rigid, local rules into a broader consensus. This approach is also your first exposure to ensemble methods, which are techniques that combine multiple simple models to create a more powerful one.