In Part 5, I drew the widest possible line between two groups. This chapter of the Hands-On Machine Learning book gets to a decision a different way: by asking one yes-or-no question after another, like a flowchart. It is the same iris flowers as before, 150 of them, and the model's job is to say which of three species a flower belongs to, from its petal length and width. The data is real, and the flowers are only a small, clear example.
A tree with two questions was enough: "is the petal length 2.45 cm or less?", then "is the petal width 1.75 cm or less?". It got 144 of the 150 flowers right, including all 50 setosas. A person can read it, which is its main attraction.
Key Points
The lesson: a decision tree is the easiest model to explain, because it is a list of questions that anyone can follow. The computer writes the questions: at each step it tries every question it could ask, keeps the one that separates the groups best, and repeats inside each answer. Its weakness is the other side of the same coin. Left alone, it keeps asking until it has memorised every training flower, and a small change in the data can change even its first question.
What I learned from it:
- Left alone, a tree keeps splitting until it has memorised the training data, which is why the tree in Part 2 scored an error of exactly 0 on its own training houses.
- Limiting how deep the tree may grow, or how few flowers may be left at the end of a branch, stops it from memorising. This is called regularization, and it is the same idea as the penalties in Part 4.
- Trees are unstable: training on slightly different data can change every question.
The chapter also covers trees that predict numbers, with each branch ending in an average, and how to draw a tree. I do not go into those here.
Quick Example
A tree, two questions deep, trained on the petal length and width of all 150 iris flowers:
PYTHON
1from sklearn.datasets import load_iris 2from sklearn.tree import DecisionTreeClassifier 3 4iris = load_iris(as_frame=True) 5X_iris = iris.data[["petal length (cm)", "petal width (cm)"]].values 6y_iris = iris.target 7 8tree_clf = DecisionTreeClassifier(max_depth=2, random_state=42) 9tree_clf.fit(X_iris, y_iris)
When I ran it, the tree learned two questions:
| Question | If yes | If no |
|---|---|---|
| Is the petal length 2.45 cm or less? | Iris setosa: all 50 setosas, and nothing else | Ask the next question |
| Is the petal width 1.75 cm or less? | Iris versicolor: 49 versicolors and 5 virginicas | Iris virginica: 45 virginicas and 1 versicolor |
For a flower with petals 5 cm long and 1.5 cm wide, it answers "no", then "yes", and says Iris versicolor, with a chance of 90.7%: 49 of the 54 training flowers in that leaf.
What training means here
Nothing is nudged, and nothing is worked out in one calculation either. Scikit-Learn grows the tree one best question at a time. At the top, it tries every measurement and every possible cut-off, such as 2.45 cm, and picks the question that splits the flowers into groups that are as nearly as possible all one species. Then it does the same inside each group, and so on, until it reaches the depth limit or the groups are pure. It never goes back to change an earlier question.
That greedy, one-decision-at-a-time growth is quick, but it does not guarantee the best possible tree, and it is why a small change in the data can change the very first question. My MiniGPT model works the opposite way: every one of its numbers is adjusted a little on every step, and nothing is ever fixed for good.
Running the notebook in Jupyter
The chapter is in the notebook that goes with the book, 06_decision_trees.ipynb. I ran everything before the exercise solutions, which took a few seconds on my Mac Studio, and then looked through the results in JupyterLab.
Graphviz drew the tree: the first question, petal length 2.45 cm or less, sets apart all 50 setosas, and the second asks about petal width
Each question cuts the petal plane with a straight line parallel to an axis: depth 0, then depth 1, and the dotted lines a deeper tree would add
For petals 5 cm long and 1.5 cm wide, the tree says 90.7% Iris versicolor and 9.3% Iris virginica
With no restrictions, the tree carves out tiny regions around single points; with min_samples_leaf = 5 it is smoother, and it scored 0.92 on new data instead of 0.898
A regression tree predicts the average value in each region, so its predictions are steps, and a deeper tree has more of them
Next, Scikit-Learn (Part 7) works through chapter 7: ensemble learning and random forests, which combine many trees.