Orthogonality and Projections

Imagine you are trying to find the perfect recipe for a cake based on past baking experiments. You have data: the amount of sugar used, the baking time, and the final taste score. The relationship isn't perfect—sometimes the cake is dry even with enough sugar—but there is a general trend. This topic gives you the mathematical tool to find that single best-trend line through scattered data points. It allows you to predict outcomes accurately by filtering out the noise, which is exactly what happens when your phone suggests the next word, when a search engine finds documents similar to your query, or when AI systems learn from vast amounts of information. Without this tool, we could only fit data perfectly on random points, which is useless for prediction.

Think of a flagpole standing on flat ground. The pole itself represents a vector—a direction and length in space. The ground represents a subspace, which you can picture as a flat room where if you draw any straight line across the floor, that entire line stays within the floor; it never breaks through the ceiling or walls. When sunlight shines down vertically, the pole casts a shadow on the floor. That shadow is the projection of the pole onto the ground. It is the part of the pole’s direction that lies entirely within the flat surface of the floor. The rest of the pole—the part sticking straight up into the air—is perpendicular to the floor and has no component lying within it. In mathematics, we often have a complex vector (like a high-dimensional description of a word or an image) and a simpler subspace (like a specific set of meaningful directions). Projecting the vector onto the subspace means finding the shadow it would cast if light shone perpendicularly from above. This is useful because it lets us approximate a complicated object by a simpler one that lives in a smaller space, keeping only the part that matters relative to our chosen directions and discarding the rest. For example, if you have two vectors (arrows), you can measure how aligned they are using a calculation called the dot product. You multiply their corresponding components together and add up the results. If the dot product is zero, the arrows are at right angles to each other—they share no common direction. This state of being perpendicular is what mathematicians call orthogonal. In machine learning, we constantly have high-dimensional data points, and we want to express them in terms of a few important directions. Projection is the tool that collapses the complex data onto these key directions, allowing us to simplify the problem while minimizing error.

Imagine you are standing in a large empty hall with a long, straight railing running across the floor. You hold a laser pointer and shine it at a point on the ceiling. The beam of light travels in a straight line from your hand to that specific spot on the ceiling. Now, imagine there is a second identical railing directly above the first one, attached to the ceiling. If you want to know which point on the upper railing corresponds most closely to your target on the ceiling, you look for the point where a line drawn perpendicular (at a ninety-degree angle) from the ceiling spot would hit the upper railing. That point is the projection of the ceiling spot onto the upper railing. The distance between your original target and this projected point represents the error or 'residual'—the part of your signal that cannot be explained by the direction of the railings. This geometric relationship holds true even when you cannot visualize it: in higher dimensions, projecting a vector onto a subspace is always finding the closest point in that subspace to the original vector, measured by the shortest possible perpendicular path.

Two vectors are orthogonal when they are at right angles to each other, meaning their dot product—the sum of the products of their corresponding components—is zero. A projection of a vector onto a subspace is the closest point within that subspace to the original vector, found by moving along the perpendicular direction until you hit the subspace.

This concept sits alongside two neighboring topics: least squares, which is the process of finding the best-fit line or plane by minimizing the sum of squared perpendicular errors between data points and the fit; and singular value decomposition (SVD), which is a method for breaking down complex matrices into simpler orthogonal components to reveal the underlying structure of data.