K-Nearest Neighbours

This method helps you make predictions based on similarity to things you already know. You will use it to: (1) label new items by finding the known items most like them, (2) estimate numbers like prices or temperatures by averaging nearby examples, (3) build a working system that stores information but does no upfront calculation, and (4) grasp the central problem of machine learning: making sure a rule works on new cases, not just the ones it has seen. After this, you will study Decision Trees, Logistic Regression, and Naive Bayes — each relies on the same core idea you learn here: how to measure how similar two things are.

Imagine you want to guess whether a new fruit is an apple or an orange. You have a box of known fruits, each with a label. You pick the new fruit, measure its weight and colour, and then look at the three or five known fruits that are most similar. If most of those are apples, you say apple. That is it — no rules about what an apple is, no formula for colour, no training. The method simply stores all the known examples and uses them at the moment of guessing.

Think of moving to a new city and trying to figure out the local food culture. You don't have a manual. Instead, you ask three or five of your closest neighbours — the people who live nearest to you. If most of them eat vegetarian, you conclude the area is vegetarian. You are not inventing a rule; you are trusting proximity. The analogy works because K-Nearest Neighbours is exactly that: the 'neighbours' are the training examples, and 'closeness' is measured by some distance. The choice of k (how many neighbours to ask) is a decision you make, like choosing to ask three people instead of ten. The analogy breaks down when you have millions of neighbours: asking millions of people is slow, and that slowness is the cost of KNN. Also, if the city has no clear majority, asking neighbours is useless — just as KNN fails when classes are heavily overlapping.

K-Nearest Neighbours (KNN) is an algorithm that classifies a new point by finding the k closest training examples and taking a majority vote of their labels, or predicts a continuous value by averaging the values of those k neighbours.

You have not yet studied any other machine learning algorithms, so this stands alone as your foundation. It is the first of several algorithms that learn from examples without building an explicit rule. Later, you will compare it with Decision Trees (which build a tree of rules), Logistic Regression (which fits a line), and Naive Bayes (which uses probabilities). KNN is different because it does no learning at all — it just stores the data. This is why it is called a 'lazy' learner, and it is the most direct illustration of the principle that all machine learning is about generalising from examples.