In Part 15, a network read a sequence of numbers one step at a time. This chapter of the Hands-On Machine Learning book gives it text, and then shows why the field moved on to a different idea, attention. It is the chapter closest to my MiniGPT model, and it ends with the Transformer, the design that MiniGPT is built on. The tasks are real: writing in the style of Shakespeare, deciding whether a movie review is positive, and translating English into Spanish.

The results in my run were mixed, partly because I shortened the longest trainings. The Shakespeare model got 54.3% of the next letters right and wrote mostly nonsense. A review model that was shown padding as if it were words scored 50.0%, no better than a coin; told to ignore the padding, it scored up to 87.5%. And a translation model that could look back at every English word beat one that could not, 61.9% to 52.9% of Spanish words right.

Key Points

The lesson: a language model is trained to predict what comes next, and writes by sampling from its predictions, which is exactly what MiniGPT does with letters. The earlier way of reading text, a network that carries a memory forward word by word, has to squeeze everything it has read into that memory. Attention is the fix: each word looks directly at every other word, scores how much each matters, and takes a weighted mix. The Transformer uses only attention, with a wave pattern added to each word so that it still knows the word order.

What I learned from it:

• Padding that is not masked out is read as text. A small fix to what the model is shown took one model from 50.0% to 87.5%.
• A translator that squeezes the whole English sentence into one summary mangles long sentences: "I like soccer and also going to the beach" became "me gusta la misma y [UNK]" ([UNK] marks a word it did not know). With attention, it became "me gusta el fútbol y voy a la playa".
• Pretrained models carry the biases of their training text. One sentiment model called "I am from India." positive and "I am from Iraq." negative.

The chapter also covers keeping several candidate translations instead of one (beam search), reading in both directions, and ready-made Transformers from Hugging Face, which I touch on below.

Quick Example

The character model is small. It looks at 39 different characters (the letters in lower case, a few punctuation marks, and the new line), turns each into 16 numbers with an embedding, reads them with a gated recurrent unit (GRU) of 128 neurons, and predicts a probability for each of the 39 possible next characters:

It learns from 1,115,394 letters of Shakespeare, in windows of 100 letters, and it is trained exactly like MiniGPT: the target for each letter is the letter that follows. Here are the results I got in my version of the chapter's runs, where I cut the longest trainings short to save time (details below):

The text that the character model wrote, after only 3 epochs, is not good. (The stateful version, after 10 epochs, wrote "to be or not to be good sorrow. / gremio: / and i have heard the great", better, with a speaker's name in the right place.) With a temperature of 0.01, which almost always picks the most likely next letter, it wrote "To be or not to be a strange / to see the duke of a strange for the se". With a temperature of 1, it wrote "To be or not to be gir? / hooks! why but beriversolvesf.". With 100 it wrote only noise. These are the same temperature settings as in my MiniGPT's demo.

What training means here

All of these models are trained as before: a loss, slopes by backpropagation, and a nudge. What is different is what the loss compares. For the character model it is the next letter, and for translation it is the next Spanish word, given the English sentence and the Spanish words so far. A model that is trained to predict the next word is a language model, and MiniGPT is one, with letters instead of words.

In a recurrent network, the information about the first word has to survive every step to the end of the sentence. Attention removes that problem. For each word, it computes a score against every other word, turns the scores into shares that sum to one, and takes a weighted mix of the other words' values. It is the same mechanism as the queries, keys, and values in my MiniGPT posts, and the Transformer in this chapter is the same design: layers of attention and small feed-forward networks, with layer normalization and skip connections from Part 14 around each. The only real difference is that MiniGPT has only a decoder, because it writes one text and does not translate. A Transformer for translation has an encoder for the English and a decoder for the Spanish.

The one new idea is the positional encoding. Attention looks at all the words at once, so by itself it cannot tell the order. The notebook adds a pattern of sine and cosine waves to each word, different for each position, so that word order is in the numbers. MiniGPT does the same job with its position embedding.

Running the notebook in Jupyter

The chapter is in the notebook that goes with the book, 16_nlp_with_rnns_and_attention.ipynb. This is the heaviest notebook in the series for my Mac. Its graphics processor, an Apple M1 Max, was no faster than its processor on small models like these, so I used the processor, and I shortened the training: the character model to 3 epochs instead of 10, and each translation model to 1 epoch instead of 10. I then ran the sentiment models and the pretrained sentence encoder again at the notebook's full length, 5 and 10 epochs. The translation results above come from the shortened runs, and the book's are better. I ran the five parts of the notebook at the same time, which took a few hours. I did not run the final fine-tuning cell with the Hugging Face Trainer, which failed to import here, and the notebook's stateful RNN, which keeps its memory from one window to the next, ran for the full 10 epochs.

Next, Scikit-Learn (Part 17) works through chapter 17: autoencoders, generative adversarial networks (GANs), and diffusion models.