Gram-Schmidt

Gram–Schmidt is the procedure that takes a messy set of directions in space and straightens them out until they are perfectly at right angles to each other and all the same length. Imagine you have a pile of sticks that are tangled and pointing in overlapping ways. You want to rearrange them so they form a perfect corner, like the three edges of a box meeting at a point, but you are not allowed to throw any information away — the new sticks must still reach every corner the old ones did. This is useful because calculations become much simpler when your reference directions don't interfere with one another. For example, if you are trying to find the best fit for a line through a cloud of data points, or if you want to compress an image without losing important details, having these 'straightened out' directions makes the math stable and precise. Without this tool, many computer graphics and data analysis systems would produce noisy or incorrect results because they wouldn't know how to separate overlapping signals.

Let us walk through a concrete example in two dimensions. Imagine you have two sticks (vectors) starting from the origin.

Stick A points to the coordinate (3, 0). It lies flat on the horizontal axis.
Stick B points to the coordinate (1, 2). It goes right 1 unit and up 2 units.

These two sticks are independent — you cannot reach Stick B’s tip by just stretching Stick A. But they are not perpendicular; they overlap.

Step 1: Keep Stick A as it is. This becomes your first new stick, call it U1. So U1 = (3, 0).

Step 2: Now fix Stick B. We want to remove any part of B that points along U1. This removal process is called "projecting". Imagine shining a light from above; the shadow that Stick B casts onto Stick U1 is the overlap.

To find this shadow (the projection), we use a simple formula: take the dot product of B and U1, divide by the length squared of U1, and multiply by U1.

The dot product of B(1, 2) and U1(3, 0) is (1*3) + (2*0) = 3.
The length squared of U1(3, 0) is 3^2 + 0^2 = 9.
So the scaling factor is 3 / 9 = 1/3.

The shadow vector is (1/3) * U1 = (1/3) * (3, 0) = (1, 0).
This means 1 unit of Stick B is "wasting" its effort pointing along the horizontal axis.

Step 3: Subtract this shadow from Stick B to get the part that is truly new and perpendicular to U1. This remainder is our second new stick, call it V2.

V2 = B - Shadow = (1, 2) - (1, 0) = (0, 2).

Check: Is V2 perpendicular to U1? U1 is (3, 0) and V2 is (0, 2). They form a right angle. One is horizontal, one is vertical. Perfect.

Step 4: Normalize. This means making the stick exactly length 1 without changing its direction.
The length of V2(0, 2) is 2.
Divide V2 by 2: U2 = (0, 1).

So, starting with messy sticks (3, 0) and (1, 2), Gram–Schmidt has produced two perfectly perpendicular, unit-length sticks: (3, 0) [which we keep as-is since it’s already on an axis] and (0, 1). Wait, actually, for U1, we usually also normalize it to be length 1. Let's do that.
Length of U1(3, 0) is 3. Normalized U1 becomes (1, 0).

Final result: The orthonormal basis is (1, 0) and (0, 1). These are the standard axes. We have straightened the messy input into the clean grid.

Why does this matter? In higher dimensions, like a machine learning model processing a word with 100 numbers, you don’t want one number to just be a scaled version of another (redundancy). Gram–Schmidt ensures each number carries unique information.

Imagine you are an architect trying to describe the orientation of a crooked picture frame hanging on a wall. The current frame is tilted, so its sides are not aligned with the vertical and horizontal edges of the room. You want to define a new set of axes that align perfectly with the room’s corners (vertical and horizontal) but still describe the exact same physical tilt.

You start with the first side of the frame, which points roughly up-right. You keep this direction as your baseline. Now, look at the second side of the frame, which should be perpendicular to the first. It isn’t quite; it leans a bit forward into the wall.

To fix this, you shine a light from the direction perpendicular to the room’s depth. The "shadow" the second side casts onto the first side represents the overlap or "leakage" between them. You subtract this shadow from the second side. What remains is the pure part of the second side that points in a truly new direction, completely at right angles to the first.

Finally, you stretch or shrink both sides so they are exactly one meter long, making them unit vectors. Now you have a clean, orthogonal coordinate system that describes the frame’s tilt without any directional confusion. The caveat is that this analogy fails if the picture was actually lying flat against the wall with no thickness — in math terms, if your starting sticks were already dependent (one was just a copy of the other), the "remaining" part would vanish completely, resulting in a zero vector, which breaks the process.

Gram–Schmidt is a method that takes any set of independent directions and transforms them into a new set where every direction is at a right angle to all others and has a length of one. It does this by taking each new direction, removing the parts of it that overlap with the directions already chosen, and then scaling the result to unit length.

This process sits directly beside "orthonormal bases" (sets of perpendicular vectors of length 1) because Gram–Schmidt is the primary way to create one from arbitrary inputs. It also connects to "linear regression" because finding the best-fit line in statistics relies on projecting data onto perpendicular axes, a step made computationally stable by the orthogonal vectors Gram–Schmidt produces.