Two-Class and Multiclass Classifiers

Everything you build on from here assumes you can tell one thing from another. Spam vs not-spam, fraud vs not-fraud, sick vs healthy — that's two-class classification, and it's the simplest case of the skill you'll use for the rest of your machine learning journey. Once you can do two, adding more classes is a pattern, not a new idea. This unlocks: Model Evaluation (how to score a classifier), Churn Prediction (will this customer leave?), Crop Disease Detection (which disease is on this leaf?), and eventually deep learning, where classification is the backbone of image and speech recognition.

Think of a line on a piece of paper. On one side, dots labelled 'A'. On the other, dots labelled 'B'. Two-class classification is drawing that line so it separates the dots as well as possible. A classifier is that line, plus a rule: 'if a new dot falls on the A side, call it A; if it falls on the B side, call it B.' Multiclass is when the paper has three or more kinds of dots. Now one line isn't enough. You can draw several lines, each one separating one kind from the rest (that's one-vs-rest), or you can draw lines between every pair of kinds (that's one-vs-one), or you can use a model that naturally handles many kinds at once, like a neural network. Each approach has trade-offs in speed, accuracy, and how confident the answers feel.

You are a fruit sorter at a market. In the morning, you sort only apples and oranges. That's two-class: you look at size, colour, and weight, and you decide apple or orange. You develop a rule: 'round and orange → orange; red and slightly pointed → apple.' Now imagine the market adds bananas, mangoes, and grapes. You can't use a single 'apple or not' rule for everything. One strategy: sort everything into 'is it an apple?' first, then 'is it an orange?', then 'is it a banana?', and so on. Anything that doesn't fit any single category goes into a 'miscellaneous' pile — that's one-vs-rest. Another strategy: compare apples to oranges, then apples to bananas, then apples to mangoes, and so on for every pair, then take a vote among all those pairwise comparisons — that's one-vs-one. A third strategy: instead of separate rules, you develop a single mental model that directly names the fruit from all its features at once. That's a natively multiclass model. The one-vs-rest approach is simple but can get confused when two fruits look similar (like a green apple and a lime). The pairwise approach is more careful but requires more comparisons, so it's slower. The natively multiclass model, if you can build one, is often the best — but it's harder to understand why it made a choice.

Two-class and multiclass classifiers are models that assign an input to one of a set of categories, where the two-class case distinguishes between exactly two categories and the multiclass case distinguishes among three or more, using strategies like one-vs-rest, one-vs-one, or native multiclass architectures.

This builds directly on your notes for Logistic Regression, which models the probability of a binary outcome. That's the two-class case. The coefficients in logistic regression are interpretable as odds ratios, and we extend that idea to multiclass by comparing each class against a reference. It also connects to Decision Trees and K-Nearest Neighbours, which naturally handle multiple classes; in fact, your notes on Decision Trees already assume multiclass. The idea of a 'line' separating classes is the same as the 'decision boundary' in Rule-Based Classifiers, but here the line is learned from data rather than hand-crafted.