In Part 2, I predicted a number, the price of a house. This chapter of the Hands-On Machine Learning book predicts a category instead, and the task is to look at a small picture of a handwritten digit and decide whether it is a 5. Reading handwritten digits is a real job, for postcodes and cheques, and the pictures come from MNIST, the standard practice set: 70,000 of them, each 28 × 28 = 784 pixels. Asking only "is this a 5?" is the book's way of starting with the simplest version of the problem.
The lesson is how to tell whether an answer like this is any good, which is less obvious than it sounds. A program that says "not a 5" for every picture is right 91% of the time, because only about one picture in ten is a 5. The model that I trained was right 95% to 96% of the time, which sounds much better, but it found only 65% of the 5s.
Key Points
The lesson: a high percentage of right answers can hide a poor model, when the answer I am looking for is rare. To see what is going on, I count four kinds of answer: 5s that it found, 5s that it missed, other digits it wrongly called 5s, and other digits that it rightly left alone. From those counts come two scores:
- Of the pictures it called 5s, how many really were 5s. For my model, 83.7%. This is called precision.
- Of the real 5s, how many it found. For my model, 65.1%. This is called recall.
The two pull against each other. If I make the model stricter about what it calls a 5, it is right more often when it says 5 (90.0%), but it misses more of them (it found only 48.0%). Which matters more depends on what the model is for.
What I learned from it:
- Compare any classifier with one that always gives the same answer, as the "not a 5" program does, before being impressed by its percentage.
- A count of every kind of right and wrong answer, called a confusion matrix, shows what the model gets wrong, which a single percentage cannot.
- The model here, a
SGDClassifier, is the first in the series trained by nudging: its numbers start at zero and are moved a little, over and over, as I explain below. SGD stands for stochastic gradient descent.
The chapter also covers a third way to compare classifiers, the ROC curve, and how to classify all ten digits at once. I do not go into those here.
Quick Example
The "is this a 5?" classifier, trained on the 60,000 training images:
PYTHON
1from sklearn.linear_model import SGDClassifier 2 3y_train_5 = (y_train == '5') # True for all 5s, False for all other digits 4 5sgd_clf = SGDClassifier(random_state=42) 6sgd_clf.fit(X_train, y_train_5)
Here is how it did when I ran the notebook, scored on training images that were held back from it in each of three rounds, as in Part 2, next to a random forest and a "classifier" that always says "not a 5":
| Model | Accuracy | Precision | Recall |
|---|---|---|---|
| Always "not a 5" | 0.91 | — | 0 |
| SGD | 0.95 to 0.96 | 0.837 | 0.651 |
| SGD, stricter | — | 0.900 | 0.480 |
| Random forest | — | 0.990 | 0.872 |
"SGD" is the SGDClassifier; "stricter" is the same model with its threshold raised until 90% of the images it calls 5s really are 5s.
The confusion matrix of the SGDClassifier, the count of every kind of answer, tells the story behind its percentage: it called 3,530 of the 5s correctly, missed 1,891, and wrongly called 687 other digits 5s.
What training means here
The SGDClassifier is the first model in this series that learns by nudging. It starts with all its numbers at zero, one weight for each of the 784 pixels plus one more, and then goes through the training images one at a time, nudging the 785 numbers a little after each image in the direction that would have made it less wrong. It passes over the 60,000 images again and again, up to 1,000 times, until the nudges stop helping. When I trained it on all ten digits with scaled pixels, it warned that it had used up those 1,000 passes before settling down: a reminder that nudging can run out of steps.
Part 4 shows this kind of training in detail. It is the same kind of step my MiniGPT model takes, but MiniGPT needs backpropagation to find its slopes, because it has many layers; a linear classifier like this one has only one, so its slope comes from a simple formula.
The chapter's other models learn as before: the random forest is grown split by split, the support vector machine (SVM) is solved directly, and k-nearest neighbors just remembers all 60,000 images.
Running the notebook in Jupyter
The chapter is in the notebook that goes with the book, 03_classification.ipynb. I ran everything before the exercise solutions, which took about 31 minutes on my Mac Studio, almost all of it in the cells that train on all 60,000 images three times over for cross-validation. Then I looked through the results in JupyterLab.
The first image in MNIST: 784 pixels, drawn as a 28 × 28 grid, and labelled as a 5
The first 100 images: some digits are clear, and some would trouble a person
The 5-detector scored 0.95 to 0.96 accuracy, but a classifier that always says "not a 5" scored 0.91
The confusion matrix: 3,530 5s found, 1,891 missed, and 687 other digits wrongly called 5s, which gives a precision of 0.837 and a recall of 0.651
Moving the threshold trades precision against recall: as one goes up, the other comes down
The random forest's precision/recall curve stays near the top right corner, well above the SGDClassifier's
The confusion matrix for all ten digits: the bright diagonal is right answers, and the column for 8 collects many of the mistakes
Next, Scikit-Learn (Part 4) works through chapter 4: how linear models are trained, including gradient descent.