Eigenvalues and Eigenvectors

This unlocks how machines find hidden structure in data. You will use it to understand Principal Component Analysis (PCA) — the technique that reduces thousands of measurements to a few meaningful ones — and to see how search engines rank pages, how recommendation systems find patterns, and why some models can be compressed to run on a phone. Without eigenvalues and eigenvectors, you cannot truly understand why PCA works, why PageRank orders results, or why certain AI systems can be shrunk without losing much. This is the gateway to multivariate statistics and optimization, which power most of machine learning.

Imagine you have a rubber sheet stretched on a frame. You can push, pull, and twist it, but you never tear it. This is a transformation — it takes every point on the sheet and moves it to a new spot. Now, on that sheet, draw a grid of arrows. After you stretch and twist the sheet, most arrows will point in a completely new direction. But a few special arrows might only get longer or shorter — they do not change direction at all. Those special directions are the eigenvectors, and the amount by which they stretch (or shrink) is the eigenvalue. If an eigenvalue is 2, the arrow doubles in length. If it is 0.5, it halves. If it is 1, it stays the same length. If it is negative, the arrow reverses direction but stays on the same line. Eigenvalues and eigenvectors are the hidden, intrinsic axes of a transformation — the directions that the transformation merely stretches, not rotates. They tell you what the transformation does at its most fundamental level, no matter how you choose to describe the sheet (the coordinate system).

Think of a master chef tasting a soup. She has a palate that recognizes individual flavors — saltiness, sweetness, sourness — and she knows that the soup can be described by how much of each of these fundamental tastes it has. The soup is a complex mixture, but she can decompose it into these basic, non-interfering dimensions. Now, stretch the analogy to a machine learning model. The data — say, measurements of a thousand customers — is the soup. The eigenvectors are the fundamental, independent 'tastes' of the data — the directions of maximum variance, where the data spreads out the most. The eigenvalues are how strong that taste is — how much variance is along that direction. PCA, which we will learn later, is the chef's palate: it finds these eigenvectors and eigenvalues to reduce a complex, high-dimensional soup to a few essential ingredients, discarding the weak flavors (small eigenvalues) that add little. This analogy stops working when you realize that, unlike a chef tasting a single soup, an eigenvector is not a property of the data itself but of the transformation (the covariance matrix) you apply to it. The chef's fundamental tastes are fixed; the eigenvectors of a data matrix are learned from the data and change with it.

An eigenvector of a square matrix is a non-zero vector that, when the matrix is applied to it, only changes in length (scales) and does not rotate; the corresponding eigenvalue is the factor by which its length changes.

This is the heart of linear algebra, which you have already studied in your library's notes on matrices and linear transformations. An eigenvector is a special case of a linear transformation — one where the output is just a scalar multiple of the input. It connects directly to your notes on 'Linear Transformations' (the matrix acts on a vector) and 'Matrix Multiplication' (applying the matrix to a vector is a matrix-vector product). It also leads to the 'Singular Value Decomposition' (SVD), which is a more general version of this idea that works for any matrix, not just square ones. And it is the mathematical engine behind 'Principal Component Analysis' (PCA), which you will explore later.