Gram-Schmidt

Gram–Schmidt is the tool that turns any set of independent directions into a perfectly perpendicular, unit-length set — an orthonormal basis. This is the foundation for understanding how least-squares regression works (the solution is a projection), how principal component analysis (PCA) finds the most important directions in your data, and why QR decomposition makes numerical calculations stable. It also appears in how language models represent the meaning of words — embeddings are often made orthogonal to remove redundancy. Without this, you can't truly grasp why the math behind many AI systems works instead of just trusting that it does.

Imagine you are standing in a field and you want to describe where you are. You could say 'go 3 steps north, then 2 steps east.' North and east are perpendicular — they don't overlap at all. But what if the only directions you knew were 'north-east' and 'east'? They are not perpendicular; they overlap. To describe your position accurately, you need directions that don't get in each other's way. Gram–Schmidt takes a set of overlapping directions (like north-east and east) and systematically builds a new set that is perfectly perpendicular (like north and east), preserving the original information. It does this one direction at a time: take the first direction as is, then for each new direction, subtract off any part that points along the directions you already have, leaving only the part that points in a totally new way. The result is a set of directions where each one is at a right angle to all the others — and if you also shrink each to length 1, you get an orthonormal basis. This process is the key to making many calculations in machine learning clean and stable.

Think of building a shelf with three sticks: one lies along the wall, one points slightly upward, and one points diagonally across the room. You want all three to be mutually perpendicular — like the edges of a box. You can't just use the sticks as they are; the diagonal one overlaps with the wall stick and the upward one. So you follow a procedure: take the wall stick as your first reference. For the second stick (the upward one), you notice it goes partly along the wall direction — so you remove that part, keeping only the part that goes directly upward. Now you have two sticks at right angles. For the third stick (the diagonal one), you remove the parts that point along the wall and along the upward direction, leaving only the part that goes in a new, perpendicular direction. Finally, you shorten each stick so it has length exactly one meter — that's normalizing. Now you have three sticks that are all at right angles to each other, each one meter long, and together they span the same space as the original three. This works in any number of dimensions, not just three. But the analogy stops working when you try to think of 'direction' in more than three dimensions — you can't picture it, but the math works the same. Also, the procedure assumes the original sticks are independent — none is a combination of the others. If one is redundant, the procedure will give you a zero-length stick, which tells you the set wasn't independent after all.

Gram–Schmidt is a procedure that takes a set of linearly independent vectors and produces an orthonormal set (each vector is perpendicular to all others and has length 1) by, one at a time, subtracting from each new vector its projection onto the already-constructed vectors, then normalizing the result.

You have not yet met vectors or dot products formally, but you have likely seen coordinates: a point (x, y) is a pair of numbers. In linear algebra, a vector is just a list of numbers, and the dot product is a way to measure how much two vectors point in the same direction. Gram–Schmidt builds on the idea of projection, which you will also meet in least squares: the best approximation of a point by a line is the perpendicular projection. The result of Gram–Schmidt is an orthonormal basis, which makes many calculations simpler — for instance, when you express a vector in terms of an orthonormal basis, the coordinates are just dot products with the basis vectors. Later, you will see that the QR decomposition (which uses Gram–Schmidt) is what makes solving least-squares problems numerically stable, and that eigenvalues and eigenvectors are related because an orthonormal basis of eigenvectors diagonalizes a matrix, exposing its intrinsic structure.