Softmax and Cross-Entropy

Every classifier in modern deep learning ends with the same two-step ritual: turn raw scores into probabilities with softmax, then measure the gap to the true label with cross-entropy. Together they form the most-used loss function on the planet — present in every image classifier, every transformer, every language-model head. The combination is not arbitrary; it is the principled outcome of fitting a categorical distribution by maximum likelihood, and the math falls out beautifully.

Given raw logits $\mathbf{z} = (z_1, \ldots, z_K) \in \mathbb{R}^K$, the softmax maps them to a probability vector: Each output is positive and they sum to one. Two intuitions are essential. First, softmax is a smooth approximation of argmax — large logits dominate the sum, near-ties get split. Second, only differences between logits matter: adding a constant to every $z_k$ changes nothing, because the constant cancels between numerator and denominator. This shift-invariance is what makes the log-sum-exp trick numerically stable: subtract $\max_j z_j$ from every logit before exponentiating to prevent overflow.

Cross-entropy measures how surprised the model is by the true label. For a one-hot target $\mathbf{y}$ with $y_c = 1$ marking the correct class, It is just the negative log-probability the model assigns to the right answer. Combined with softmax, the loss for a single example reduces to $\ell(\mathbf{z}, c) = -z_c + \log \sum_j e^{z_j}$ — the log-sum-exp form, which is convex in $\mathbf{z}$ and stable to differentiate.

Now the magic. The gradient of softmax-plus-cross-entropy with respect to the logits is the cleanest formula in machine learning: Predicted minus true. That is it. Every quadratic term that you would expect from differentiating a softmax cancels against the cross-entropy. This is *why* the pair is universal: backprop through this final layer costs one subtraction per logit, and the gradient is exactly the prediction error in probability space.

From an information-theory angle, minimizing cross-entropy is equivalent to minimizing the Kullback-Leibler divergence $\mathrm{KL}(p_{\text{data}} \,\|\, p_{\text{model}})$ between the empirical distribution and the model. KL is always non-negative and zero only when distributions match — so cross-entropy training pushes the model's probabilities toward the data-generating distribution. From a Bayesian angle, it is maximum likelihood under a categorical observation model. Three viewpoints, one objective.

Worked example — softmax on three logits: $\mathbf{z} = (2, 1, 0)$. Exponentiate: $(e^2, e^1, e^0) \approx (7.389, 2.718, 1.000)$. Sum: $11.107$. Softmax: $(0.665, 0.245, 0.090)$. Add the same constant to every logit, say $-2$, giving $(0, -1, -2)$ — the softmax is identical. Now subtract the max for stability: $(0, -1, -2)$ → $(e^0, e^{-1}, e^{-2}) = (1, 0.368, 0.135)$, sum $1.503$, softmax $(0.665, 0.245, 0.090)$. ✓