Mathematics for Machine Learning

Introduction and Motivation

Machine learning is, at its core, a mathematical conversation with data. When you train a neural network, fit a regression model, or cluster…

• loss landscapes
• gradient descent
• model fitting

Linear Algebra

A system of linear equations is a collection of equations in which each unknown appears only to the first power and is never multiplied by a…

• MLPs
• embeddings
• linear layers

Analytic Geometry

A norm on a vector space is a function that assigns each vector a non-negative length. Formally, it must satisfy three axioms: absolute homo…

• k-NN
• cosine similarity
• kernel methods

Matrix Decompositions

The determinant of a square matrix is a single scalar that captures the signed volume scaling factor of the linear map . If , the map triple…

• PCA
• recommender systems
• low-rank attention

Vector Calculus

The derivative of a function measures its instantaneous rate of change at . Formally, . Geometrically, it is the slope of the tangent line t…

• backprop
• autograd
• neural training

Probability and Distributions

A probability space is a mathematical stage for reasoning about uncertainty. It consists of three ingredients: a sample space (the set of al…

• softmax
• Bayesian inference
• calibration

Continuous Optimization

Gradient descent is the simplest and most-used continuous optimization algorithm: to minimize a differentiable , start at some and iterative…

• SGD
• Adam
• Lagrangian methods

When Models Meet Data

The previous seven chapters built a mathematical toolbox. Chapter 8 turns the key and opens the door to machine learning proper. Three ingre…

• train/test split
• regularization
• cross-validation

Linear Regression

Linear regression is the first and most important supervised learning algorithm. The model asserts that the target is a linear function of f…

• OLS
• ridge
• Bayesian linear regression

Principal Component Analysis

Principal Component Analysis (PCA) is the most-used dimensionality reduction technique and a perfect payoff for everything learned in linear…

• PCA
• whitening
• eigenfaces

Gaussian Mixture Models

A Gaussian Mixture Model (GMM) assumes the data is generated by underlying clusters, each Gaussian with its own mean and covariance: where a…

• GMM
• EM algorithm
• soft clustering

Support Vector Machines

Support Vector Machines (SVMs) solve binary classification by finding the maximum-margin hyperplane that separates two classes. Given labele…

• SVM
• kernel SVM
• hinge loss

Deep Learning Foundations

Every classifier in modern deep learning ends with the same two-step ritual: turn raw scores into probabilities with softmax, then measure t…

• MLPs
• activation functions
• backpropagation

Deep Learning Building Blocks

A convolution slides a learnable filter across an input feature map and outputs, at each spatial position, the dot product between the filte…

• cross-entropy loss
• KL divergence
• InfoNCE

Generative Models

A VAE is a generative model that learns to compress data into a continuous latent space and decode latents back into data. Two networks: an…

• stable softmax
• mixed precision
• well-conditioned linear systems

Reinforcement Learning

A Markov Decision Process (MDP) formalizes sequential decision-making. It's the tuple : state space , action space , transition probabilitie…

• bias-variance
• VC dimension
• PAC bounds

Classical ML & Evaluation

A decision tree is a sequence of axis-aligned splits that partitions feature space into rectangles. At each leaf the model predicts the majo…

• VAE
• Stable Diffusion
• DDPM

Markov Chains and Monte Carlo

A Markov chain is a memoryless random walk on a state space — and its stationary distribution is the universal target of MCMC, the convergen…

• MCMC
• HMC / NUTS
• particle filters

Fourier Analysis and Tensor Algebra

Any reasonable periodic function can be written as a sum of sines and cosines — the Fourier series. This single change of basis powers JPEG,…

• FFT / spectrograms
• Whisper
• multi-head attention

Statistics & Optimization Foundations

Hypothesis tests and confidence intervals turn finite samples into rigorous statements about populations — the same machinery that powers A/…

• bootstrap CIs
• L-BFGS
• natural gradient

Python for ML: Code the Math

Set up Python, meet NumPy, and learn the one rule that makes machine-learning code fast: vectorize everything. Every cell in this chapter ru…

• NumPy
• Pyodide in-browser
• PCA from scratch

Probabilistic Graphical Models

A Bayesian network factorizes a giant joint distribution into a directed graph of small conditional tables — turning an exponential problem…

Gaussian Processes & Kernels

A kernel is a similarity function that secretly computes an inner product in a (possibly infinite-dimensional) feature space — letting linea…

Causal Inference

A hidden common cause can make two variables move together even when neither causes the other — confounding is why observational correlation…

Divergences, Line Search & Bayesian Approximation

KL divergence is asymmetric and can blow up to infinity. The Jensen-Shannon divergence symmetrizes it and stays bounded by log 2 — one membe…

Probability & Linear-Algebra Foundations You Reuse

Six identities — linearity of expectation, Var(aX+b)=a²Var(X), the variance-of-a-sum rule, and the covariance/correlation definitions — do 9…

PCA Revisited: Projection and Latent Variables

The second face of PCA: project data onto a low-dimensional subspace so that the reconstruction error is as small as possible — and discover…

The Mathematics of Model Adaptation

Fine-tune a billion-parameter model by training two thin matrices per layer: the rank factorization ΔW = BA turns a 16M-number update into 6…