Image Segmentation

Image segmentation allows a computer system to understand *what* is in an image down to the smallest detail, rather than just knowing that certain things are present. This precision is critical for tasks where edges matter. For example, in medical imaging, identifying the exact boundaries of a tumour can determine whether surgery is possible. In autonomous driving, knowing the precise edge of the road or the shape of an obstacle prevents accidents caused by vague estimations. In photo editing, it allows you to remove a background seamlessly without leaving jagged artifacts around a person. It turns a rough estimate into a usable, pixel-perfect map of reality.

When you look at a photograph, your eye instantly distinguishes between different parts of the scene. If there is a cat sitting on a sofa, your brain knows exactly which points belong to the cat and which belong to the sofa. Image segmentation does this digitally: it assigns a specific category to every single pixel in the image.

Consider a photo with 10 million pixels. A simpler approach, called object detection, might draw a rough rectangle around the cat. This rectangle says, 'there is a cat somewhere here,' but it also includes parts of the sofa and the floor behind the cat. Segmentation goes further. It draws a precise outline that follows the curve of the cat’s whiskers, its tail, and its ears. Every pixel inside that outline is labeled 'cat'; every pixel outside is labeled 'sofa' or 'floor.'

This level of detail comes with a cost. Creating training data for this system is much harder than for simple object detection. If you use the rectangle method, you only need to draw one box per cat. With segmentation, someone must manually label every single pixel in that box as 'cat' and every pixel outside as 'not cat.' In an image with millions of pixels, this is a time-consuming job.

Imagine a teacher taking attendance in a large lecture hall. A bounding-box approach (object detection) is like the teacher glancing at the room and saying, 'There are 50 students here,' while drawing a vague circle around the entire group of people. The teacher knows roughly where they are but doesn't know who is who or exactly where one person ends and another begins.

Image segmentation is like the teacher reading out every seat number in the hall. For seat 1A, they say 'occupied.' For seat 1B, they say 'empty.' They assign a specific status to every single individual spot. This allows the teacher to know exactly who is present and where they are sitting.

The analogy breaks down in one key way: in a classroom, a person occupies a distinct, solid space. In an image, pixels can blend together. A single pixel might be on the edge of a blue shirt and a white collar, making it hard to assign it purely to one category without 'blurring' the line. The segmentation system must handle these soft edges carefully.

Image segmentation is the process of labeling every individual pixel in an image with a specific category, such as 'road,' 'car,' or 'sky,' creating a precise map of objects and backgrounds rather than just detecting their rough locations.

Two common variations exist: one which labels all pixels of the same type together (e.g., all cars are labeled 'car' regardless of which car it is), and another which distinguishes between individual objects of the same type (e.g., labeling 'car #1,' 'car #2,' etc.), as well as combining both to identify every distinct object and its surrounding background in a single view.

You have previously learned that neural networks can classify entire images, deciding if a picture is generally of a cat or a dog. Segmentation extends this by applying classification to tiny regions within the image instead of the whole picture. This concept works closely with object detection, which finds rough locations (bounding boxes) but lacks pixel-level detail, and with instance segmentation, which adds the ability to distinguish between multiple individual objects of the same class, such as separating two cars parked next to each other.