19 · Decision tree classification: entropy, Gini & info gain
Build a tree that recursively asks binary questions about features — each split chosen to maximize how much purer the subsets become.
A decision tree recursively splits the feature space along single-feature thresholds — at each split it picks the question that purifies the resulting subsets most.
Without this:
Without decision trees you lose the building block of random forests, gradient boosting, and XGBoost — the most successful classical-ML algorithm family.
A decision tree answers a classification question by asking a sequence of yes/no questions about individual features. Each internal node poses a threshold test — "is petal_length ≤ 2.45?" — and routes examples left or right depending on the answer. Leaf nodes predict a class.
Greedy recursive splitting
The tree is grown greedily: at each node, try every (feature, threshold) pair and pick the one that reduces impurity the most. After the split, recurse on each child. Stop when nodes are pure, or a stopping criterion is met (max_depth, min_samples_split, etc.).
Impurity measures
Two dominant criteria for measuring how "mixed" a node is:
Gini impurity (sklearn default):
Gini(t) = 1 − Σ pₖ²
where pₖ is the fraction of class k at node t. A pure node (all one class) has Gini = 0. A 50/50 binary split has Gini = 0.5.
Entropy (information-theoretic):
H(t) = −Σ pₖ log₂ pₖ
Same intuition: 0 for a pure node, 1 for a 50/50 binary split.
Information gain
The gain of a split is the reduction in impurity from parent to weighted children:
IG = Impurity(parent) − [|left|/|parent| · Impurity(left) + |right|/|parent| · Impurity(right)]
The algorithm picks the (feature, threshold) pair that maximises IG.
Categorical vs numeric features
For numeric features sklearn scans all midpoints between consecutive sorted values as candidate thresholds. For categorical features with k categories, sklearn one-hot-encodes them internally — the "threshold" becomes "is feature == category?".
sklearn API
DecisionTreeClassifier(criterion='gini') is the default. Key hyperparameters: max_depth, min_samples_split, min_samples_leaf, max_leaf_nodes.
Python (in browser)
`plot_tree` renders the full learned tree. Each node shows the split condition (feature ≤ threshold), gini impurity, sample count, per-class distribution, and the majority class. The root split on petal_length (≤ 2.45) perfectly isolates all setosa examples in one leaf — matching the well-known fact that setosa is linearly separable from the other two species.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Gini and entropy are interchangeable in practice — they produce near-identical tree structures on real datasets. Both peak at maximum uncertainty (p=0.5 for binary) and reach 0 for a pure node. Information gain (IG) measures how much a split reduces parent impurity; the tree picks the split maximising IG at each node.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The most visually striking property of decision trees: the decision boundary is always axis-aligned — a mosaic of rectangles. Each rectangle corresponds to one leaf of the tree. The non-smooth, staircase shape is a direct consequence of splitting on a single feature at a time. Contrast with an SVM RBF kernel (smooth elliptic curves) or logistic regression (a single diagonal line).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`model.feature_importances_` gives each feature a score in [0, 1] summing to 1. The score equals the total weighted Gini reduction from all splits that used that feature. For Iris, petal dimensions dominate because the root split and the second-level splits all use petal_length or petal_width. This is one of the simplest built-in feature selection signals in sklearn.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Gini is the default for a reason: it avoids a log evaluation at every split, which adds up across the thousands of candidate splits evaluated during tree growth. On large datasets or deep trees this matters. Entropy is worth trying if you want interpretable information-gain values or if the slight pruning difference affects your downstream use case.
If a node has 10 examples of class A and 0 of class B, what is its Gini impurity?
- A decision tree greedily picks the (feature, threshold) split that maximises information gain (reduction in Gini or entropy) at each node. It recurses until leaves are pure or a stopping criterion fires.
- Gini = 1 − Σ pₖ²; entropy = −Σ pₖ log₂ pₖ. Both are 0 for a pure node and peak at maximum disorder. In practice they produce nearly identical trees; Gini is the sklearn default.
- Decision boundaries are always axis-aligned rectangles — one feature split at a time. Diagonal separators require many splits to approximate.
- No feature scaling required — trees split on raw values, so mixed units and scales are handled natively.
- An unconstrained tree will perfectly overfit training data. Always set max_depth (or another stopping knob) to generalise.
Sklearn's `DecisionTreeClassifier` is rarely used alone in production but is the atomic unit of random forests, gradient boosting, and XGBoost — by far the most-deployed tabular ML algorithms.
If you remove it: You'd lose the building block of ensemble methods — which dominate tabular ML competitions.