Gini and Information Gain

This unlocks Decision Trees, the first model in your library that you can read like a rulebook. From there you'll move to Random Forests and Gradient Boosting — the algorithms that win most machine-learning competitions on tabular data. You'll also need this before Churn Prediction and Crop Disease Detection, since both rely on trees. Understanding how a tree decides where to split is the difference between trusting a model and understanding it.

Imagine you're sorting a box of mixed fruit — apples and oranges — into two trays. You can't see inside the box, but you can feel each fruit. Your goal: after one feel, put as many of the same fruit on each tray as possible. Before you feel anything, the box is a messy mix. After you sort, each tray is cleaner — mostly apples here, mostly oranges there. The 'messiness' of a tray is what we call impurity. Gini impurity and information gain are two different tape measures for that mess. A good split is one that makes each tray much less messy than the original box. Gini asks: 'If I pick a fruit at random from this tray and then guess its kind by the majority in the tray, how often will I be wrong?' Entropy (used in information gain) asks: 'How much surprise is there in this tray?' Both give you a number: 0 means perfectly clean, higher numbers mean messier. The tree tries every possible split and picks the one that gives the biggest drop in mess — that drop is the 'information gain'.

Think of a teacher sorting students into two study groups based on one yes/no question, trying to separate those who will pass an exam from those who won't. The teacher can ask: 'Did you sleep more than 6 hours?' or 'Do you have a study plan?' or 'Do you own a pet?' The teacher doesn't know the answers yet, but she has last year's data. She computes: for each question, if I split the students by the answer, how pure would each group be? A question that perfectly separates pass/fail gives pure groups — no mess. A question that leaves both groups half pass, half fail gives no gain — the split is useless. The teacher picks the question with the biggest gain. This works because impurity is measurable: it's the opposite of 'purity', a number from 0 (all same) to some maximum (perfectly mixed). Where does this analogy break? In reality, a split rarely gives pure groups, and the tree often needs many splits to reach useful purity. Also, the tree doesn't ask a question once — it recurses, asking new questions on each subgroup, building a tree. And the tree doesn't choose questions that are 'meaningful' to humans; it simply optimises the arithmetic of impurity. So the teacher analogy is a clean story, but the real tree is a relentless number-cruncher that will split on any feature if it reduces mess, even if the question seems silly to us.

Gini impurity and entropy-based information gain are two mathematical measures of how mixed (impure) the classes are within a node of a decision tree, used to decide which feature and threshold to split on at each step.

This is part of Machine Learning Foundations, under the umbrella of learning functions from data. You've already met Decision Trees as a model family (interpretable, but prone to overfitting). This concept gives you the precise arithmetic inside a tree's split decision. It also connects to K-Nearest Neighbours — both are non-parametric, meaning they don't learn fixed coefficients, but trees use impurity to partition space, not distance. And this leads to Decision Jungles, a compressed variant that also uses impurity measures to choose merges.