Random Forests

Random forests are one of the most reliable tools in machine learning. They give you a strong baseline for almost any tabular problem, tell you which features matter, and rarely require delicate tuning. Learning this unlocks practical modelling on real datasets, and prepares you for the decision jungles and deep forests used in resource-constrained settings. It also builds intuition for ensemble methods, a key idea behind modern gradient boosting and even large language models.

A single decision tree can be unstable: a tiny change in the training data can change the whole tree structure, making its predictions jump around. Random forests fix that by building many trees and averaging their votes. Each tree is trained on a random sample of the data (with replacement) and considers only a random subset of features at each split. This randomness makes the trees different, and the averaging smooths out the noise, giving a more reliable and often more accurate prediction than any single tree.

Think of a committee making a decision, like a group of doctors diagnosing a patient. If you asked just one doctor, their diagnosis might depend heavily on which training they had, which medical journal they read last, or even the time of day. But if you assemble a diverse panel and ask each doctor to vote, the group decision is usually more balanced and robust. Random forest is that panel: each tree is a doctor, given a random subset of the patient's symptoms (features) and a random subset of past cases (data). The final diagnosis is the majority vote. Where the analogy breaks: if all doctors were trained identically, they'd all make the same mistakes. Random forests avoid this by shuffling the data and features, so each tree sees a slightly different world. Also, the panel cannot explain its reasoning as clearly as one doctor, so random forests trade interpretability for accuracy.

A random forest is an ensemble of decision trees, each trained on a bootstrap sample and using random feature subsets at each split, whose predictions are averaged (for regression) or majority-voted (for classification).

You have notes on Decision Trees, which are the building blocks here. A random forest is essentially many decision trees, but with two crucial tweaks: each tree gets a different random sample of the data (bootstrap) and, at each split, it only considers a random subset of features. This makes the trees diverse, so averaging their predictions reduces variance. Compare with K-Nearest Neighbours, which makes predictions based on closest examples; random forests combine many local rules. This is also your first taste of ensemble methods, a theme that continues in advanced models.