Dimensionality Reduction and PCA

This unlocks the machinery behind embeddings, visualising high-dimensional data, and compressing datasets so models train faster and generalise better. It is your gateway to understanding how LLMs represent meaning, how recommendation systems condense user preferences, and how anomaly detection simplifies signals. Concretely, after this you can: read a covariance matrix, interpret a scree plot, apply PCA to a real dataset in Python, and explain why scaling matters — skills that directly underlie the embedding layers of every transformer.

Imagine you are a librarian in a vast, dusty archive. You have thousands of documents, each described by hundreds of vague attributes—word counts, sentence lengths, unusual punctuation. You cannot see the forest for the trees. Dimensionality reduction is like finding the few true axes along which the documents actually differ: the 'formality' axis, the 'technicality' axis, the 'emotional tone' axis. These axes are not the original features; they are combinations of them. PCA finds the directions in which your data spreads out most—the directions that carry the most information. If you project your data onto these directions, you lose some fine detail but capture the essential shape of the cloud.

Think of a shadow. A three-dimensional object casts a two-dimensional shadow on a wall. That shadow is a projection: it flattens one dimension but preserves enough shape to recognise the object from a particular angle. Now imagine you can rotate the object to find the angle where its shadow is most informative—where the silhouette shows the most variation, the most distinguishing features. PCA does exactly that for your data: it finds the 'best angle' (the principal components) to cast a lower-dimensional shadow that captures the greatest spread of the data. Where the analogy breaks: a shadow preserves only what the light touches—it cannot capture depth behind the object. PCA similarly loses information along the discarded dimensions. But unlike a shadow, PCA is not a single fixed view; it is the optimal view for that data. Also, a shadow retains the raw geometry of the object, whereas PCA's components are blends of all original features—sometimes making them harder to interpret.

Principal Component Analysis (PCA) is a dimensionality reduction technique that projects data onto the directions of greatest variance—the principal components—which are linear combinations of the original features, sorted by how much variance they capture, so that you can represent the data with fewer numbers while retaining most of its structure.

You are building this from scratch. However, this sits inside the unsupervised learning family, alongside the clustering methods you will study later (k-means, DBSCAN, hierarchical). It is the compression half of unsupervised learning, where clustering is the grouping half. PCA also connects forward to the embedding layers of neural networks—those are learned, nonlinear versions of what PCA does linearly. Your library notes from 'Linear Algebra for Machine Learning Workbook' stress that eigenvectors of the covariance matrix are the heart of PCA, and that scaling is non-negotiable.