2 · PCA: intuition and math
Find the orthogonal axes of maximum variance — and use them to compress features without losing the signal.
PCA finds the orthogonal directions of maximum variance in the data — the first few capture most of the signal, the rest are dropped.
Without this:
Without PCA you'd lack the universal first-pass dimensionality reduction algorithm and the conceptual basis for SVD-based methods.
Principal Component Analysis (PCA) is the workhorse of linear dimensionality reduction. Given a matrix X (n samples × d features), PCA finds k orthogonal directions — the principal components — along which the data spreads the most. Project X onto those k directions and you have a compressed representation that preserves as much variance as possible.
The mathematical recipe:
- Center the data: subtract the column mean. Scale if features are in different units (scale-sensitive!).
- Covariance matrix: Σ = (1/n) · XᵀX — a d×d matrix where Σᵢⱼ captures how features i and j co-vary.
- Eigendecompose: Σ = VΛVᵀ — eigenvalues λᵢ are the variances along each PC direction; eigenvectors vᵢ (columns of V) are the PC directions.
- Sort: largest λ first — the first PC captures the most variance.
- Project: Z = X · V_k (the top k eigenvectors), yielding an n×k matrix.
Explained variance ratio: λᵢ / Σλ is the fraction of total variance captured by the i-th PC. Plot the cumulative version to pick k — a common heuristic is "keep enough PCs to explain 90% of variance".
Why SVD? sklearn's PCA uses SVD of X (not eigendecomp of Σ) for numerical stability. SVD never needs to explicitly form Σ, which can be ill-conditioned. We'll unpack the equivalence in the next lesson.
Python (in browser)
The two principal components of correlated 2D data — PC1 aligns with the main diagonal (max variance), PC2 is perpendicular.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Scree plot — the 'elbow' shows how quickly variance is captured. 90% of variance in the breast cancer dataset is captured by just 7-10 PCs out of 30.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Accuracy on 2 PCA components is surprisingly close to using all 30 features — PCA preserves the discriminative signal.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Breast cancer data projected to first 2 PCs — the two classes already separate without using any label information.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The SVD ↔ PCA equivalence — understanding this lets you read sklearn's components_ and explained_variance_ without confusion.
Singular Value Decomposition — the numerical engine inside sklearn's PCA. Understanding SVD makes the components_ attribute and the reconstruction formula click.
Why must you scale features before applying PCA?
- PCA finds orthogonal directions (principal components) of maximum variance by eigendecomposing the covariance matrix Σ = XᵀX/n.
- Always StandardScale before PCA — a feature in a large numeric range will dominate the covariance structure and distort all components.
- The explained variance ratio λᵢ/Σλ tells you what fraction of total variance each PC captures — use a cumulative scree plot to pick k.
- sklearn implements PCA via SVD of X (not eigendecomp of Σ) — more numerically stable, especially for wide matrices.
- PCA is class-unaware — if discriminative directions are low-variance, prefer LDA when labels are available.
Image compression (eigenfaces), gene expression analysis, NLP word embedding visualisation via PCA-to-2D, exploratory data analysis on any high-D dataset.
If you remove it: You'd lack the canonical compressor and the entry point to all SVD-based techniques — from eigenfaces to latent semantic analysis to matrix factorisation in recommendation systems.