K-Nearest Neighbours

This is your first complete machine learning algorithm. It is the easiest to understand and the hardest to fool yourself with. You will use it to: (1) classify new data points by looking at their nearest known examples, (2) predict continuous values like prices or temperatures, (3) build a working system with no training phase at all, and (4) understand the entire challenge of machine learning: how to make a model that works on new data, not just on the data it has seen. After K-Nearest Neighbours, you will study Decision Trees, Logistic Regression, and Naive Bayes — each of these builds on the core idea you learn here: how to measure similarity.

Imagine you want to guess whether a new fruit is an apple or an orange. You have a box of known fruits, each with a label. You pick the new fruit, measure its weight and colour, and then look at the three or five known fruits that are most similar. If most of those are apples, you say apple. That is it — no rules about what an apple is, no formula for colour, no training. The method simply stores all the known examples and uses them at the moment of guessing.

Think of moving to a new city and trying to figure out the local food culture. You don't have a manual. Instead, you ask three or five of your closest neighbours — the people who live nearest to you. If most of them eat vegetarian, you conclude the area is vegetarian. You are not inventing a rule; you are trusting proximity. The analogy works because K-Nearest Neighbours is exactly that: the 'neighbours' are the training examples, and 'closeness' is measured by some distance. The choice of k (how many neighbours to ask) is a decision you make, like choosing to ask three people instead of ten. The analogy breaks down when you have millions of neighbours: asking millions of people is slow, and that slowness is the cost of KNN. Also, if the city has no clear majority, asking neighbours is useless — just as KNN fails when classes are heavily overlapping.

K-Nearest Neighbours (KNN) is an algorithm that classifies a new point by finding the k closest training examples and taking a majority vote of their labels, or predicts a continuous value by averaging the values of those k neighbours.

You have not yet studied any other machine learning algorithms, so this stands alone as your foundation. It is the first of several algorithms that learn from examples without building an explicit rule. Later, you will compare it with Decision Trees (which build a tree of rules), Logistic Regression (which fits a line), and Naive Bayes (which uses probabilities). KNN is different because it does no learning at all — it just stores the data. This is why it is called a 'lazy' learner, and it is the most direct illustration of the principle that all machine learning is about generalising from examples.