I read Aurélien Géron's Hands-On Machine Learning chapter by chapter, ran its notebooks on my Mac, and wrote up what I saw. This is the short version, for anyone who has not read the nineteen posts: what I learned, what I would still use, and what I would skip. It has no notebook of its own. Everything here comes from the posts, which have the numbers.
The main lessons
- Judge a model on data it has not seen. A model can memorise what it learned from and look perfect. A decision tree scored an error of $0 on its own training houses, and $67,153 on new ones (Part 2).
- Compare with a simple guess first. A program that says "not a 5" for every handwritten digit is right 91% of the time, and "the same as last week" was hard to beat when forecasting train boardings (Part 3, Part 15).
- Most of machine learning is one idea. Start with random numbers, work out which way each should move to be less wrong, move it a little, and repeat. The rest of the series is where the direction comes from, and what "wrong" means.
- Fixes can be small and decisive. Telling a review model to ignore padding took it from 50.0% to 87.5% (Part 16).
- Training is fragile. A picture-drawing model drew one shirt 32 times, and agents reached the top score and then fell away (Part 17, Part 18).
How every model learned
There turned out to be only four ways to find a model's numbers.
| Way of learning | Parts | What happens |
|---|---|---|
| Worked out exactly | 1, 2, 5, 8 | A formula or a solver gives the answer in one calculation, such as the best straight line through points. |
| Remembered | 1 | The model stores its training data, and does its work when it is asked a question. |
| Grown | 6, 7 | A flowchart of yes-or-no questions is built one question at a time, and groups of them vote. |
| Nudged | 3, 4, 9, 10 to 19 | Every number is moved a little in the direction that makes the answers less wrong, many times over. |
Nudging first appears in Part 3, and is explained in Part 4. From Part 10 on, almost everything is nudged. For a straight line, the direction comes from a formula. For a network with many layers, it comes from a method called backpropagation. What "wrong" means changes too: for a classifier it compares the answer with a label, for a picture-copying network the picture with itself, for a language model the next letter with the real one, which is what my MiniGPT does, and for reinforcement learning the value of an action with the reward that followed it.
What I would still use
I wrote "still used?" answers in several parts, and they add up to a picture. This is my reading, from what I ran, and from what I know of the field.
| Still a good first choice | Why |
|---|---|
| The workflow of Part 2 | Setting aside a test set, trying a simple baseline, and scoring on held-back data matter for every model. |
| The scores of Part 3 | Precision and recall show what accuracy alone hides. |
| Nudging and its settings (Part 4) | Everything after it is built on it. |
| Forests of trees, and boosting (Parts 6 and 7) | For tables of data, they are often the best tool. |
| Squeezing data and finding groups (Parts 8 and 9) | Everyday tools for looking at data and making it smaller. |
| The standard tricks for deep networks (Part 11) | They are the defaults of modern libraries. |
| Retraining someone else's network (Parts 11 and 14) | It beat training from scratch in my runs. |
| Attention (Part 16) | The design behind today's language models. |
| Good to understand, less likely to use | Why |
|---|---|
| The widest-street method of Part 5 | Niche now, though the picture is a good one. |
| Several methods for unrolling curved data (Part 8) | Mostly historical. |
| TensorFlow loops, data pipelines, and serving (Parts 12, 13, and 19) | The ideas carry over, but much of the field has moved to PyTorch. |
| Networks that carry a memory along a sequence (Parts 15 and 16) | Attention replaced them for language. |
| Networks that forge pictures against a detective (Part 17) | Diffusion has largely taken over. |
The book was not padding. It is a survey, and a survey includes what was current when it was written. But I would not spend equal time on every chapter.
What I would do differently
I would run the long trainings at full length from the start. In Part 16 and Part 17 I first shortened the longest ones to save time, and the results were poor: a picture-drawing model that made one picture, and another that made dark blurs. Rerunning at full length fixed most of that, but it took hours, and I did not rerun the translation models. My Mac has a graphics processor, and I tried it: for these small models, with batches of 32, it was slower than the processor, as Part 12 shows. A bigger machine, or bigger batches, is where a graphics processor would help.
I did write the training loop of Part 12 in PyTorch, as well as in TensorFlow, because the loop that I wrote by hand in MiniGPT is in PyTorch. The two gave the same results.