Orthogonality and Projections

This unlocks the entire machinery of linear algebra in machine learning. You will see projections when you fit lines to data (least squares), when you search for similar documents (RAG embeddings), when you compress models for phones (PCA/SVD), when you stabilise numerical calculations (orthogonal matrices), and when you build Gram-Schmidt to create orthonormal bases. Master this and every later concept—backpropagation, optimisation, attention—will rest on ground you already know.

Think of a flagpole casting a shadow on the ground. The shadow is the projection of the pole onto the ground—the part that lies flat. The pole is perpendicular (orthogonal) to the ground, and the shadow vanishes when the sun is directly overhead and the pole is vertical. In mathematics, a vector is like a pole, and a subspace (like the ground) is a flat surface. The projection of the vector onto the subspace is the shadow it would cast if light shone perpendicularly. This is useful because it lets us 'approximate' a vector by a simpler one that lives in a smaller space, keeping the part that matters and discarding the rest. In machine learning, we do this constantly: we have high-dimensional data, and we want to express it in terms of a few important directions. Projection is the tool that does that.

Imagine you are in a room with a wall. You hold a laser pointer that shines a beam perfectly horizontally (perpendicular to the wall). You point the laser at the wall from the floor. The spot where the beam hits the wall is the projection of the laser pointer's tip onto the wall. If you raise the pointer, the spot moves up; if you move it sideways, the spot moves sideways. The laser always hits the wall at a right angle. That perpendicularity is the key. Now, if you have a vector (like the direction and distance from you to something), you can project it onto a line or plane (like the wall) by dropping a perpendicular. The projection is the closest point on that line or plane. This is exactly what least squares does: it finds the line that makes the vertical (perpendicular) errors as small as possible. The analogy breaks down when you have more than three dimensions: you can't picture a 'wall' in a 1000-dimensional space, but the math still works. The dot product measures perpendicularity: if it's zero, the vectors are orthogonal, meaning one has no 'wall' component.

Two vectors are orthogonal when their dot product is zero, meaning they are perpendicular. A projection of one vector onto another is the shadow it casts when light shines perpendicularly from above—computed as (a·b / b·b) * b.

You have no prior maths beyond arithmetic, so this page starts from zero: what a vector is, what a dot product is, and how to add and multiply numbers. Once you master this, you'll see that least squares (the normal equations) is a projection, Gram-Schmidt is repeated projections, eigenvalues and eigenvectors often come in orthogonal pairs, and SVD is built on orthogonal matrices. This is the foundation for everything that follows.