In Part 6, I grew one decision tree, and saw how unstable it is. This chapter of the Hands-On Machine Learning book asks what happens if I train many models and let them decide together, which is called an ensemble. The random forest that won in Part 2 is exactly this, a vote of many decision trees.
The example is made up: 500 points in two interlocking half-moon shapes, with noise, so that no straight line can separate them. It is the kind of problem where a single model is only fairly good. The best single model I tried was right on 89.6% of the test points. Three different models voting together were right on 91.2% to 92.0%, and a model that learns how best to combine them reached 92.8%.
Key Points
The lesson: a group of different models, each only fairly good, can beat every one of its members, because their mistakes are different and cancel out. There are three ways to build a group: have the models vote; train many copies of one kind of model, each on a different random sample of the data (called bagging, and a random forest is this, with decision trees); or train models one after another, each correcting the mistakes of the ones before it, which is called boosting, shown in a screenshot below, and often gives the best results.
What I learned from it:
- Randomness helps: trees trained on different samples make different mistakes, and averaging them cancels many of the mistakes out.
- A forest also reports how much each fact helped it, which is a cheap way to see what matters.
- Boosting builds its group in order, so it cannot be trained in parallel.
The chapter also covers a final model that learns how to combine the others (stacking), which appears in the table below, and a way to test each tree on the samples it never saw.
Quick Example
Three different classifiers, voting on the moons dataset, 500 points in two interleaved half-circles:
PYTHON
1from sklearn.datasets import make_moons 2from sklearn.ensemble import RandomForestClassifier, VotingClassifier 3from sklearn.linear_model import LogisticRegression 4from sklearn.model_selection import train_test_split 5from sklearn.svm import SVC 6 7X, y = make_moons(n_samples=500, noise=0.30, random_state=42) 8X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42) 9 10voting_clf = VotingClassifier( 11 estimators=[ 12 ('lr', LogisticRegression(random_state=42)), 13 ('rf', RandomForestClassifier(random_state=42)), 14 ('svc', SVC(random_state=42)) 15 ] 16) 17voting_clf.fit(X_train, y_train)
Here is how each model did on the test set when I ran the notebook. The score is the share of test points classified correctly:
| Model | Test accuracy |
|---|---|
| Logistic regression, on its own | 0.864 |
| Random forest, on its own | 0.896 |
| SVM (support vector machine), on its own | 0.896 |
| The three voting: the majority wins | 0.912 |
| The three voting: their chances are averaged | 0.92 |
| 500 bagged decision trees, each trained on its own random sample of the training set | 0.92 |
| Stacking: a random forest that learns from the three models' predictions | 0.928 |
What training means here
An ensemble does not bring a new way of training; it trains many models in the ways earlier parts have shown, and combines them. Bagging grows 500 decision trees, each on its own random sample of the training points, one split at a time. Voting trains three different models side by side, and stacking then trains one more on their predictions.
Gradient boosting comes closest to nudging. It starts with one small tree, then trains each new tree on the errors the trees before it left behind, and adds a fraction of that tree's answer to the total: a step downhill on the error, which is where the "gradient" in the name comes from. But it steps by adding whole trees, each grown split by split, rather than by adjusting numbers, the way my MiniGPT model does.
Running the notebook in Jupyter
The chapter is in the notebook that goes with the book, 07_ensemble_learning_and_random_forests.ipynb. I ran everything before the exercise solutions, which took about a minute on my Mac Studio, and then looked through the results in JupyterLab.
I trained three different classifiers on the moons: on their own they score 0.864, 0.896 and 0.896
One decision tree carves an overfitted boundary; 500 trees, each trained on a random 100 points and voting together, draw a smoother one
A random forest measures how much each feature helps: petal length 0.44 and petal width 0.42, while the sepals hardly matter
Gradient boosting: each new tree is trained on the errors the earlier trees left behind, and adding the trees together fits the data better and better
Stacking trains a final random forest on the three classifiers' predictions, and scored 0.928, the best in the chapter
Next, Scikit-Learn (Part 8) works through chapter 8: dimensionality reduction, making data smaller while keeping what matters.