Every MiniGPT so far has learned the same way, the guessing game from Part 2: guess the next token, check the real one, and nudge the parameters. Meta's small Llama 3.2 models learned from something more. The Llama 3.2 announcement says that "logits from the Llama 3.1 8B and 70B models were used as targets" during pre-training. Logits are the scores against the rows of lm_head, just before they become a wheel of chances: the small models learned by copying what the big models thought, not just from the text. This post tries the same idea on a Mac, with a small MiniGPT as the student and four different teachers.

The code is in part6-distillation/, with a follow-along notebook for a Mac, minigpt_follow_along_6.ipynb.

The big picture, in plain English

Learning from the answer, or from a teacher

In the guessing game, the only thing the machine learns from at each step is the one right answer. Bone gets all the credit, and ball, treat, hug, and banana are all equally wrong. But they are not equally wrong: ball, treat, and hug would all make sense, and banana would be strange.

A trained model knows that. Before it writes, it has a whole wheel of chances, a slice for every token. Distillation trains the student to copy that whole wheel, not just to pick the right answer. Each guess then teaches it much more: which near misses were reasonable, and which were absurd.

But a wheel is only as good as the reading behind it. Here are two real ones. GPT-2, which learned from text on the web, gives hug 6.2%, big 2.1%, good 2.0%, and treat 1.0%, and bone just 0.03%: 501st place. Three things push it down. On the web, people give their dogs hugs, rides, runs, and new leashes more often than bones. The wheel chooses the next piece, not the next noun, so much of the chance goes on words that start a longer phrase, such as a big… or a good…. And GPT-2's wheel is spread over everything on the web: its 20 biggest slices hold only 28% between them. TinyStories-33M, a small model trained on children's stories, and one of the teachers below, puts bone first, at 24.7%, and its 20 biggest slices hold 78%. Keep that difference in mind; it turns out to matter.

The two parts of the score

The student is scored on two things at once, and both are surprise scores from Part 2:

• How surprised it is by the real next token, as before.
• How different its wheel is from the teacher's wheel. The measure is called KL divergence: 0 when the two wheels are identical, and larger the more they differ.

I weight the two equally. Before comparing the wheels, I soften both with a temperature of 2, the same kind of setting as in Part 1's step 5, so that the thin slices, the near misses, are big enough to learn from.

The teacher must use the same pieces

There is a catch, and it rules out every current big model. Comparing two wheels slice by slice only makes sense if slice 1,000 means the same token on both wheels. So the teacher must use exactly the student's tokeniser, its supply of pieces from Part 3.

My student uses GPT-2's 50,257 pieces. Qwen3 uses its own, about 151,000; Llama 3 uses another, 128,000; Gemma, Mistral, and Phi each have their own. None of their wheels line up with my student's. So the teachers here all use GPT-2's pieces. That is a bigger family than GPT-2 alone: GPT-Neo, and the models released with the TinyStories paper, use them too.

Four teachers, and no teacher

The student is the MLX machine from Part 4, with the original 2017 block, but on GPT-2's pieces instead of my 8k ones, so its token embeddings make it 30 million numbers, as in Part 3's table. I trained it five times, once with no teacher and once with each teacher, 3,000 steps each, on the same stories. Every score in the table is bits per byte on the test stories, so lower is better. The "teacher's own" column scores each teacher on its own, as a storyteller; the "student's" column scores the student it taught:

GPT-2 and GPT-2 XL learned from text on the web. TinyStories-33M is the largest model released with the TinyStories paper, trained on the whole TinyStories collection: far more stories than my slice of it. My MiniGPT-512 is a wider MiniGPT that I trained on the same practice stories as the student, for 5,000 steps.

What the teachers taught

• A bigger web-text teacher barely helped. GPT-2 XL is twelve times the size of GPT-2, and moved the student from 0.7469 to 0.7382, for four and a half times the training time. The third column says why: at 1.5 billion numbers, GPT-2 XL is still worse at these stories (0.799) than the 30-million-number student trained on them (0.756). It had never read anything like them. Even GPT-2's own gain over no teacher, 0.009, is about the size of the luck between two random starts (see Part 5), so it may be no gain at all.
• A teacher who knew the stories helped a lot. TinyStories-33M is 45 times smaller than GPT-2 XL, but it had read far more stories of exactly this kind, and it scores 0.467. It pulled the student to 0.7204, past both GPT-2s, in 23 minutes.
• The most helpful teacher was not the best storyteller. On its own, my MiniGPT-512 scores 0.644, so it is a clearly worse storyteller than TinyStories-33M, at 0.467. Yet it was the most helpful teacher by far. Its student scored 0.6936, the best of all five: 0.062 better than the student with no teacher, almost twice the 0.035 that TinyStories-33M managed. It also learned faster: it passed the no-teacher student's final score at about step 1,700, rather than step 3,000.

So why did the weaker storyteller teach better? I had a guess, and then I tested it.

Why the weaker teacher helped more

My guess was this. In distillation, the student is marked on how closely its wheel matches the teacher's, slice by slice, at every position, and it can only make the kinds of wheel its own design can produce. MiniGPT-512 is the same design as the student, only wider, and learned from exactly the same practice stories, so its wheels should be ones the student can copy. TinyStories-33M is built differently and learned from far more stories, so some of the detail in its wheels may depend on things the student cannot work out. Researchers have seen this in other distillation experiments, and call it the capacity gap (Cho and Hariharan; Mirzadeh and others): a stronger teacher does not always make a stronger student.

To test it, I trained MiniGPT-512 again, then the student three times (with MiniGPT-512, with TinyStories-33M, and with no teacher), kept all three students, and measured how far each one's wheels ended up from each teacher's, on the same test stories. The distance is the KL divergence from earlier, without the softening: 0 would mean identical wheels. Retraining gave almost the same scores as before: 0.6913 with MiniGPT-512, 0.7198 with TinyStories-33M, and 0.7554 alone, all inside the band of luck:

• Before any teaching, the student was already much closer to MiniGPT-512. With no teacher at all, its wheels were less than half as far from MiniGPT-512's as from TinyStories-33M's, and picked the same top slice 70.5% of the time, against 58.8%.
• It could copy MiniGPT-512, and mostly could not copy TinyStories-33M. Learning from MiniGPT-512 almost halved the distance to it, from 0.46 to 0.25. Learning from TinyStories-33M cut the distance to it by only about a fifth, from 1.23 to 0.96.
• The student taught by TinyStories-33M still ended up nearer MiniGPT-512's wheels (0.47) than its own teacher's (0.96). Most of what the better storyteller knew was out of this student's reach.

So the guess holds up: the most helpful teacher is the one whose wheels the student can actually reach. Each student is a single training run, so a different random start would move these distances a little, but it would not turn a gap of four times the other way.

What they write

The no-teacher student loses the thread: "he wanted to take his melon home from his eyes", a second character also called Tim, and a melon that talks. The taught student holds a scene: Tim, a friend called Sam, a lost cake, a search, and an ending, finishing with "the moral of the story is…", the closing line many TinyStories use, which the no-teacher student never picked up.

The jargon decoder

The terms for the whole series are collected in one table, the series glossary.

The code, in the order it runs

The code is in part6-distillation/. It reuses Part 4's MLX machine and Part 3's stories, with GPT-2's pieces.

The teachers: train_distill.py's --teacher

--teacher none trains the student alone. --teacher gpt2 and --teacher gpt2-xl load a frozen GPT-2 through mlx-lm. --teacher torch:roneneldan/TinyStories-33M loads TinyStories-33M in PyTorch, because mlx-lm cannot load its design, and hands its wheels across to the MLX student. And --teacher runs/teacher.safetensors loads my MiniGPT-512, which train_teacher.py trains first:

The score: in train_distill.py

Both machines see the same batch; the teacher's wheel is fixed, and only the student's parameters move:

ce is the usual surprise at the real next token. With --alpha 0.5, the two parts count equally, and --alpha 1.0 switches the teacher off altogether.

Scoring the teachers: eval_teacher.py, and testing the capacity gap: measure_gap.py

eval_teacher.py --teacher gpt2-xl measures a teacher's own bits per byte on the test stories: the "teacher's own" column in the table.

run_gap.sh retrains MiniGPT-512 and the three students, keeping their checkpoints, and then measure_gap.py compares every student's wheel with every teacher's, one batch of test stories at a time:

Try it yourself

• The follow-along notebook: part6-distillation/minigpt_follow_along_6.ipynb, for Jupyter on a Mac with Apple Silicon. It shows a teacher's wheel next to the one-hot answer, works the KL divergence out by hand, trains a student with GPT-2 as its teacher, and shows the capacity-gap results. It is saved with the outputs from my own run, so you can read every result on GitHub without a Mac.

• On the command line, after Part 3's prepare_data.py and tokenizers_setup.py:

MLX needs Apple Silicon. The GPT-2 teachers download from Hugging Face the first time.

Part 7 goes back to Part 5's modern machine and gives attention a sliding window, so that it can read a much longer row in the same memory.