Image Segmentation

Segmentation is the difference between a system that sees *where* something is and one that sees *exactly what is there*. It unlocks precise tasks like medical image analysis (tumour boundaries), autonomous driving (lane and obstacle edges), and photo editing (selecting a person precisely). It is the foundation for later study of instance and panoptic segmentation, and it is the step that turns a bounding box into a usable mask — which is what makes modern AI systems able to interact with the world rather than just pointing at it.

When you look at a photo, your eye doesn't just notice a cat — it knows which pixels are cat and which are sofa. That is segmentation: every single pixel gets a label. It is not a rectangle around the cat; it is a precise outline that follows whiskers and tail. This matters because a bounding box often includes background — think of the chair in front of the cat — so a rectangle says 'some cat here', but a mask says 'these exact pixels make up the cat'. Segmentation is classification at the pixel level, repeated millions of times per image, and that repetition is what makes it far more expensive than object detection to create training data: every pixel needs a label, not just one box per object.

Imagine you are a school teacher taking attendance. A bounding-box approach would be like counting how many children are in the classroom — you know there are 20, and you could draw a circle around the group, but you don't know exactly which child is which. Segmentation is like calling out each name and having that child raise their hand — you assign every seat (pixel) to a specific student (class). The annotation cost is much higher: you have to check every single seat, not just count heads. And the same is true in segmentation: for every pixel in every training image, someone has to say 'this is road, this is car, this is sky'. Where the analogy stops working: in the classroom, children are separate objects — you can't have half a child. But a pixel can be part of a car and part of a road in its little corner, so segmentation sometimes needs soft boundaries or overlap. Also, a child raises a whole hand, but a pixel can only carry one label in most segmentation tasks — that's the difference between semantic and instance segmentation.

Image segmentation is the task of assigning a class label to every pixel of an image, producing a pixel-level mask that delineates precise object boundaries, as opposed to a coarse bounding box.

You have just learned that a neural network can classify images — decide whether the whole image is a cat or dog. Segmentation is the natural extension: instead of one label for the whole image, we produce a label for each pixel. This pushes the concept of 'features' from the entire picture down to local regions, and it is the bridge to object detection (where you predict bounding boxes) and instance segmentation (where you also separate different instances of the same class).