MiniGPT, grown in code you can read

Neil Haddley • October 11, 2026

An experiment: the code that trains MiniGPT from random numbers, rewritten for people, treating the whole readable model as a black box, with the surprise score and the AdamW nudge written out, and a check against the original

AI Pythongpttransformerspytorchreadable-codemachine-learning

Why I rewrote the training code

In MiniGPT, rewritten to be read, I rewrote the code that runs my trained MiniGPT model, to be as easy for a person to read as I could make it. I call that rewrite the readable model. This post does the same for the code that trains it, from MiniGPT (Part 2): it starts from 826,433 random numbers and nudges them until the model writes like my exhibit.

This is what training does, in one paragraph and one picture:

Training does not need to understand the model. It takes snippets of real text, and asks the model, for every letter, what comes next. It looks up the chance the model gave the letter that really came next, and turns those chances into one number, the surprise score: small when the model expected the right letters, big when it did not. Then PyTorch works backwards through every calculation the model made, and finds, for each of the 826,433 numbers, which way it should move to make the surprise a little smaller. Every number moves a tiny step that way, and the whole thing repeats, 3,000 times. Nobody tells the model what to learn: the numbers change only because each change made the model a little less surprised by Shakespeare. I can check every step of that, but, as with the readable model, I cannot fully explain why the numbers it ends up with work.

One training step, with real numbers: the first eight practice questions of the text, and the chance the model gave each right answer, before training and after it.

Part 1 drew the model as a closed black box, and training can treat it the same way. Here is one step of training, from the outside:

Scroll sideways, or tap the diagram to open it full size

One training step, with real numbers for the first practice snippet. Training reads the chances that come out of the box, and sends changed numbers back in; it never looks inside.

So I wrote the training program to use the whole of the readable model as a black box. Was that possible? Almost. Training needs exactly three things from the box:

The list of tables, and their shapes, so that it can fill each one with random numbers to start.
A guess for every letter of a snippet, not just the last one, because every letter is a free practice question. The readable model's scores_for_next_letter keeps only the last row, so this is the one place where the training program calls pieces from inside the box: the same steps, without the [-1].
That every guess is made with ordinary PyTorch arithmetic, so that PyTorch can work backwards through it to find the slopes. The readable model already is.

A fourth would be needed to train exactly as Part 2 did: a training mode, a switch that makes the box use dropout. The readable model has none, for a good reason (see below), so this program trains without dropout.

Here is one training step as a picture. The dark boxes are the readable model; everything else is the training program; and the numbered arrows are the only three connections between them.

Nothing else about the model matters to training: not attention, not the MLP, not how many blocks there are.

The program is readable/readable_training.py in the series repo, next to the readable model's, with a notebook version saved with its outputs: open it in Colab, or read it on GitHub.

There are no Dumb Questions

Q: Why does the readable model have no dropout?

A: Because dropout only does anything during training. While a model learns, dropout sets a random tenth of the values passing through it to 0 on every step, and scales the rest up to make up for them, so that the model can never rely on any one value being there. That makes memorising the practice text harder. Once the model is trained and writing, dropout is switched off: the original code does that with model.eval(). The readable model only runs the trained model, so dropout would have been code that never does anything, and I left it out.

Q: Then why not add it for training?

A: It would mean giving the box a training mode, and the point of this program is to use the box unchanged. It would also not bring back the exhibit exactly: that would need every random draw made in exactly the same order as the original code, which would tie the readable program to the original's structure.

Q: Did training without it make things worse?

A: Not here. Both models I grew without dropout scored slightly better on the exam than the exhibit grown with it (section 12). The small model was not memorising its practice text after 3,000 steps, so dropout had little to prevent. Part 2's bigger model is different: its exam score got worse after step 1,500 while its practice score kept improving, which is exactly what dropout is for.

The rules I followed

The same rules as in MiniGPT, rewritten to be read: every value gets its own name, every shape is written down, every library call is explained, and where nobody really knows why something works, I say so.

There is one exception, and it is the point of training. The model's numbers must change, step after step, and the model must see the changes. So the nudge in section 8 is the one place in the program that changes numbers in place: it works out the new values under their own name, shrunk_table - nudge, and then copies them into the table with table.copy_(...).

Words I use

Part 2 introduced most of these, in plain words first. Here is each one, with the name experts use in brackets:

In this post, and what experts call it	What it means
snippet (a sequence; 32 of them make a batch)	128 letters in a row, from a random place in the practice text
practice text, exam text (training set, validation set)	the first 90% of the text, which the model learns from, and the last 10%, which it never sees until the exam
surprise score (loss, cross-entropy)	the average of −log(chance the model gave the right next letter)
slope (all the slopes together are the gradient)	how much the surprise would rise if one number rose a tiny bit
working backwards (backpropagation)	how PyTorch finds every slope in one go
the nudge rule (an optimiser; here AdamW)	how far, and which way, to move each number
step size (learning rate)	roughly the most a number moves on one step: 0.0003
shrinking (weight decay)	moving every number very slightly towards 0 on every step
dropout (the same)	during training only, setting some of the values passing through the model to 0 at random; not used here

The top level: one training step

Here is the whole of training on one screen. one_training_step is what the loop in section 11 runs 3,000 times, and surprise_score is the one number that everything is learned from. Everything else in the program is either a setting, a check, or one of the pieces these two call.

Scroll sideways, or tap the diagram to open it full size

one_training_step and surprise_score, with each note pointing at the code it explains. Steps 1 to 4 are the same as in section 9's picture.

1. The settings

A person chose all of these. They are Part 2's settings for growing the exhibit.

PYTHON

1TRAINING_STEPS = 3000                # how many times every number is nudged
2SNIPPETS_PER_STEP = 32               # each step practises on 32 snippets of text...
3LETTERS_PER_SNIPPET = 128            # ...each 128 letters long, the most the model can see
4SHARE_OF_TEXT_FOR_PRACTICE = 0.9     # the first 90% of the text is for practice; the last 10% is the exam
5
6STARTING_SPREAD = 0.02               # random starting weights are typically this size (as in the original)
7STEP_SIZE = 0.0003                   # how far each nudge goes: the "learning rate"
8
9# The nudge rule, AdamW, has four more settings. These are PyTorch's defaults, which the original uses.
10SLOPE_MEMORY = 0.9                   # how much of its running average slope a number keeps each step
11SQUARED_SLOPE_MEMORY = 0.999         # the same, for its running average of slope × slope
12TINY_NUMBER_TO_AVOID_DIVIDING_BY_ZERO = 1e-8
13SHRINK_RATE = 0.01                   # every number is also shrunk towards 0 a little: "weight decay"
14
15STARTING_NUMBERS_SEED = 42           # fixing the random choices makes the whole run repeatable
16SNIPPET_CHOICE_SEED = 1337
17CHECKPOINTS = [0, 100, 300, 1000, 3000]   # the steps at which the program reports how it is doing
18
19# The tables are small, so splitting each calculation between several processor cores costs more time
20# than it saves. On my Mac, one core was the fastest: 0.23 seconds a step, against 0.35 with all ten.
21torch.set_num_threads(1)

2. The black box

The program first fetches the readable model program, readable_minigpt.py, if it is not already here (in Colab, only this notebook is), and imports it under the name box. Importing a Python file runs it, and that program prints a page of examples as it runs, so contextlib.redirect_stdout sends its printing into an io.StringIO, a piece of memory that nobody reads.

Importing it also loads the trained exhibit model, as box.model. Training never uses its numbers, only their names and shapes. The exhibit appears again only in checks and comparisons.

PYTHON

1def open_the_black_box():
2    """Import the readable model program, without its printing."""
3    with contextlib.redirect_stdout(io.StringIO()):
4        return importlib.import_module("readable_minigpt")

3. The text: practice and exam

About 1.1 million letters of Shakespeare's plays, the text the exhibit was trained on. box.letters_to_ids turns it into one long list of letter IDs, and torch.tensor makes that list a table of numbers.

The first 90% is for practice. The last 10% is kept back as an exam: the model never practises on it, so its score on the exam shows whether it has learned something general, rather than memorised its practice text.

PYTHON

1TEXT_URL = "https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt"
2TEXT_FILE = Path("tiny_shakespeare.txt")
3if not TEXT_FILE.exists():
4    urllib.request.urlretrieve(TEXT_URL, TEXT_FILE)
5
6all_text = TEXT_FILE.read_text(encoding="utf-8")
7all_letter_ids = torch.tensor(box.letters_to_ids(all_text))          # [1,115,394]
8end_of_practice = int(SHARE_OF_TEXT_FOR_PRACTICE * len(all_letter_ids))
9practice_ids = all_letter_ids[:end_of_practice]                       # [1,003,854]
10exam_ids = all_letter_ids[end_of_practice:]                           # [111,540]
11
12print(f"letters of text: {len(all_letter_ids):,}  practice: {len(practice_ids):,}  exam: {len(exam_ids):,}")
13print("it begins:", repr(all_text[:60]))

CODE

1letters of text: 1,115,394  practice: 1,003,854  exam: 111,540
2it begins: 'First Citizen:\nBefore we proceed any further, hear me speak.'

4. Random starting numbers

The new model has the same tables as the exhibit, with the same names and shapes; only the numbers are new. Each table starts in one of three ways, as in the original code:

stretches (the normalising steps' ln...weight tables) start at 1, and shifts and biases start at 0. Those values change nothing: they let the normalising and the weights do their plain job.
weights and embeddings start as small random numbers. torch.randn draws numbers from the bell curve, with an average of 0 and a typical size of 1; multiplying by 0.02 makes their typical size 0.02.

PYTHON

1def starting_numbers(table_name: str, shape: torch.Size, random_generator: torch.Generator) -> torch.Tensor:
2    if "ln" in table_name and table_name.endswith(".weight"):   # a stretch
3        return torch.ones(shape)
4    if table_name.endswith(".bias"):                            # a shift or a bias
5        return torch.zeros(shape)
6    return torch.randn(shape, generator=random_generator) * STARTING_SPREAD   # a weight or an embedding

requires_grad_(True) asks PyTorch to record every calculation each table takes part in from now on, so that it can work out its slopes later. box.name_the_stored_tables then gives the tables the readable model's descriptive names. It does not copy them: the model and the dictionary of tables hold the very same tables, so nudging one nudges the other.

CODE

1numbers to grow: 826,433
2saved the grown numbers to grown.pt

5. Guesses for every letter

PYTHON

1def scores_for_every_letter(letter_ids: torch.Tensor, model) -> torch.Tensor:
2    """[n] IDs -> [n, 65]: the model's scores for what comes after each letter, not only the last."""
3    starting_hidden_states = box.first_hidden_states(letter_ids, model)               # [n, 128]
4    hidden_states_after_each_block = box.run_all_blocks(starting_hidden_states, model)
5    final_hidden_states = box.normalise(hidden_states_after_each_block[-1], model.final_stretch, model.final_shift)
6    return final_hidden_states @ model.next_letter_rows.T + model.next_letter_biases   # [n, 65]

This is the one function that reaches inside the box. It is the readable model's scores_for_next_letter without the [-1]: it keeps the scores for every position, because training uses every one of them. A check that I did not change anything: on the trained exhibit, its last row matches the readable model's answer for goo, to within rounding.

CODE

1biggest difference from the readable model's scores for the letter after 'goo': 9.5e-07

6. The surprise score

PYTHON

1def surprise_score(letter_ids: torch.Tensor, right_next_letter_ids: torch.Tensor, model) -> torch.Tensor:
2    """[n] IDs and their [n] right answers -> one number: the average surprise."""
3    scores = scores_for_every_letter(letter_ids, model)                       # [n, 65]
4    chances = box.scores_to_shares(scores)                                    # [n, 65], each row adds up to 1
5    positions = torch.arange(len(letter_ids))                                 # [n]: 0, 1, 2, ...
6    chance_of_the_right_letter = chances[positions, right_next_letter_ids]    # [n]
7    surprise_at_each_position = -torch.log(chance_of_the_right_letter)        # [n]
8    return surprise_at_each_position.mean()

For each practice question, I look up the chance the model gave the right answer. A chance of 1 means no surprise at all, a surprise of 0. The smaller the chance, the bigger the surprise: -torch.log(chance) gives 0 for a chance of 1, 4.17 for a chance of 1 in 65, and 13.8 for 1 in a million. The surprise score of a snippet is the average of its 128 surprises.

chances[positions, right_letter_ids] picks one number from each row: from row 0, the chance at the column of the first right answer; from row 1, the chance at the column of the second; and so on.

Before training, the model is a little worse than a blind guess; the exhibit is far better. PyTorch has this whole calculation as one call, cross_entropy, and it agrees:

CODE

1a blind guess (1 in 65) would score:  4.17
2the random new model scores:          4.23
3the trained exhibit scores:           1.19
4PyTorch's cross_entropy for the exhibit: 1.19
5exam score of the random new model: 4.21
6exam score of the trained exhibit:  1.69

7. The slopes: working backwards

For each of the 826,433 numbers, training needs its slope: if this one number went up a tiny bit, would the surprise score go up or down, and by how much?

surprise.backward() works out all of them in one go. While the model made its guesses, PyTorch recorded every calculation that involved a table marked with requires_grad. backward goes through that record in reverse, from the surprise score back to every table, using the chain rule from calculus at each step. It puts the answers in each table's .grad: a table of slopes, the same shape as the table itself.

This is the second black box in this program, and the one I find hardest to follow by intuition. Part 2 opens it up with real numbers. Here, rather than trust it, I tested it: I nudged one number by hand, measured how much the surprise really changed, and compared that with what its slope predicted.

PYTHON

1next_letter_biases = model_being_grown.next_letter_biases   # [65]; I test one of them: the bias for " "
2space = box.LETTER_TO_ID[" "]                                  # the space, the commonest letter in the text
3
4surprise_before = surprise_score(first_snippet, its_answers, model_being_grown)
5surprise_before.backward()                                     # work out every slope
6slope_of_the_space_bias = next_letter_biases.grad[space].item()
7
8tiny_nudge = 0.001
9with torch.no_grad():                                          # no_grad: do not record this calculation
10    nudged_biases = next_letter_biases.clone()
11    nudged_biases[space] = nudged_biases[space] + tiny_nudge
12    nudged_model = box.name_the_stored_tables({**tables_being_grown, "lm_head.bias": nudged_biases})
13    surprise_after = surprise_score(first_snippet, its_answers, nudged_model)
14
15print(f"the slope predicts a change of:  {slope_of_the_space_bias * tiny_nudge:+.7f}")
16print(f"nudging by hand changes it by:   {(surprise_after - surprise_before).item():+.7f}")
17for table in tables_being_grown.values():
18    table.grad = None                                          # tidy up: this was only a test

CODE

1the slope predicts a change of:  -0.0001068
2nudging by hand changes it by:   -0.0001068

They agree. A slope only describes a tiny nudge, because the surprise does not change in a straight line, and that is why training takes thousands of small steps rather than one big one.

8. The nudge: AdamW, written out

The simplest rule would be to move every number a little way downhill, against its slope. Part 2's notebook uses a cleverer rule, AdamW, which PyTorch provides as one call, torch.optim.AdamW. Here it is written out. For every number, it keeps two running averages from step to step:

the average slope: which way the number has been pushed lately. This smooths out the noise, because each step sees only 32 snippets.
the average squared slope: how big its pushes have been lately, whichever way.

PYTHON

1class NudgeMemory:
2    """AdamW's running averages: one table of each, the same shape as every model table."""
3
4    def __init__(self, tables: dict):
5        self.steps_taken = 0
6        self.average_slope = {name: torch.zeros_like(table) for name, table in tables.items()}
7        self.average_squared_slope = {name: torch.zeros_like(table) for name, table in tables.items()}

PYTHON

1def nudge_every_number(tables: dict, memory: NudgeMemory) -> None:
2    """One AdamW step: move every number a little, against its slope. Changes the tables in place."""
3    memory.steps_taken = memory.steps_taken + 1
4    early_correction_for_slope = 1 - SLOPE_MEMORY ** memory.steps_taken
5    early_correction_for_squared_slope = 1 - SQUARED_SLOPE_MEMORY ** memory.steps_taken
6    with torch.no_grad():
7        for name, table in tables.items():
8            slope = table.grad
9            new_average_slope = SLOPE_MEMORY * memory.average_slope[name] + (1 - SLOPE_MEMORY) * slope
10            new_average_squared_slope = (SQUARED_SLOPE_MEMORY * memory.average_squared_slope[name]
11                                         + (1 - SQUARED_SLOPE_MEMORY) * slope * slope)
12            memory.average_slope[name] = new_average_slope
13            memory.average_squared_slope[name] = new_average_squared_slope
14
15            corrected_average_slope = new_average_slope / early_correction_for_slope
16            corrected_average_squared_slope = new_average_squared_slope / early_correction_for_squared_slope
17            typical_slope_size = torch.sqrt(corrected_average_squared_slope) + TINY_NUMBER_TO_AVOID_DIVIDING_BY_ZERO
18
19            shrunk_table = table * (1 - STEP_SIZE * SHRINK_RATE)                      # weight decay
20            nudge = STEP_SIZE * corrected_average_slope / typical_slope_size          # at most about STEP_SIZE
21            table.copy_(shrunk_table - nudge)                                         # the one in-place change

Here is the same rule for one number, on one step:

The nudge is the average slope divided by the typical size of the pushes. So a number whose slopes keep pointing the same way moves about the step size, 0.0003, whether those slopes are big or small; a number whose slopes keep changing direction moves less. Every number is also shrunk very slightly towards 0: the "W" in AdamW, weight decay. Both averages start at 0, which makes them too small in the first few steps; dividing by 1 - memory ** steps_taken corrects that, and matters less and less as the steps go on.

with torch.no_grad() tells PyTorch not to record the nudge: it is not part of the model's arithmetic. table.copy_(new_values) writes the new values into the existing table, so that the model, which holds that very table, sees them.

9. One step, and the exam

PYTHON

1def pick_snippets(random_generator: torch.Generator) -> list:
2    """32 random snippets of practice text, each with its answers: the same text, one letter on."""
3    last_possible_start = len(practice_ids) - LETTERS_PER_SNIPPET - 1
4    starts = torch.randint(last_possible_start, (SNIPPETS_PER_STEP,), generator=random_generator)
5    return [(practice_ids[start:start + LETTERS_PER_SNIPPET], practice_ids[start + 1:start + LETTERS_PER_SNIPPET + 1])
6            for start in starts.tolist()]

PYTHON

1def one_training_step(model, tables: dict, memory: NudgeMemory, random_generator: torch.Generator) -> float:
2    snippets = pick_snippets(random_generator)
3    surprise_of_each_snippet = [surprise_score(letters, answers, model) for letters, answers in snippets]
4    average_surprise = torch.stack(surprise_of_each_snippet).mean()
5    for table in tables.values():
6        table.grad = None                    # forget the last step's slopes
7    average_surprise.backward()              # find this step's slopes
8    nudge_every_number(tables, memory)       # and nudge
9    return average_surprise.item()

Here is one step, in order. The dark box lists the readable model's own functions that it calls: training uses only these four, and never opens the rest.

One step picks 32 snippets at random places in the practice text, works out the average surprise over all 32 × 128 practice questions, clears the old slopes, finds the new ones, and nudges every number. torch.randint picks the 32 starting places, and torch.stack turns the 32 surprise scores into one table, so that .mean() can average them.

PYTHON

1def exam_score(model) -> float:
2    surprise_of_each_snippet = []
3    with torch.no_grad():
4        for start in range(0, len(exam_ids) - LETTERS_PER_SNIPPET - 1, LETTERS_PER_SNIPPET):
5            letters = exam_ids[start:start + LETTERS_PER_SNIPPET]
6            answers = exam_ids[start + 1:start + LETTERS_PER_SNIPPET + 1]
7            surprise_of_each_snippet.append(surprise_score(letters, answers, model).item())
8    return sum(surprise_of_each_snippet) / len(surprise_of_each_snippet)

The exam goes through the whole exam text in snippets of 128 letters, one after another, and averages the surprise. torch.no_grad() again: the exam must not teach the model anything.

CODE

1exam score of the random new model: 4.21
2exam score of the trained exhibit:  1.69

10. Checking against the original

Before the long run, I checked that the readable training does the same arithmetic as Part 2's. The program builds the original notebook's model with the same random starting numbers, and trains a copy of each for three steps on the same snippets: the original with torch.optim.AdamW, mine with the code above. Then it compares all 826,433 numbers. To compare like with like, the check switches the original's dropout off.

CODE

1after 3 steps, biggest difference in any of the 826,433 numbers: 5.2e-06
2numbers that differ by more than 0.00001: 0

The differences are rounding: the two programs do the same arithmetic in a slightly different order.

11. Growing it

Here is the run itself: a report for the checkpoints, then a short loop, because every piece of it has a name:

PYTHON

1d = box.LETTER_TO_ID["d"]
2
3
4def report(step: int, seconds: float) -> None:
5    with torch.no_grad():
6        chances = box.chances_from_scores(box.scores_for_next_letter("goo", model_being_grown),
7                                          temperature=1.0, keep_biggest=65)
8        g_row = model_being_grown.token_embedding_table[box.LETTER_TO_ID["g"], :4].tolist()
9        writing = box.write("ROMEO:", 150, model_being_grown, temperature=0.8, seed=5)
10    print(f"--- step {step:,} ({seconds / 60:.1f} minutes): exam score {exam_score(model_being_grown):.2f}; "
11          f"'g' begins {', '.join(f'{v:.3f}' for v in g_row)}; chance of 'd' after 'goo' {chances[d].item():.1%}")
12    print(writing, "\n", flush=True)
13
14
15nudge_memory = NudgeMemory(tables_being_grown)
16snippet_generator = torch.Generator().manual_seed(SNIPPET_CHOICE_SEED)
17started = time.time()
18for step in range(TRAINING_STEPS + 1):
19    if step in CHECKPOINTS or step == TRAINING_STEPS:
20        report(step, time.time() - started)
21    if step == TRAINING_STEPS:
22        break
23    one_training_step(model_being_grown, tables_being_grown, nudge_memory, snippet_generator)
24
25torch.save({name: table.detach() for name, table in tables_being_grown.items()}, "grown.pt")
26print("saved the grown numbers to grown.pt")

Five times along the way, the program reports the exam score, the start of the g token embedding, the chance of d after goo, and 150 letters written from ROMEO: by the readable model's own write:

Training step	Exam score	The `g` token embedding begins	Chance of `d` after `goo`
0, all random	4.21	−0.021, −0.003, 0.023, 0.012, …	1.4%
100	2.62	−0.019, −0.012, 0.021, 0.021, …	3.3%
300	2.39	−0.022, −0.016, 0.023, 0.027, …	4.5%
1,000	1.97	−0.030, −0.025, 0.033, 0.039, …	25.6%
3,000, finished	1.66	−0.034, −0.040, 0.042, 0.036, …	32.4%

What it writes from ROMEO:, at each checkpoint:

After no steps:

CODE

1ROMEO:o-uou-;k!zcV'djayd&veE!ky,WgFJ;D$Snly$PyWK$irstZqJuQODkfo?By?AJK:sWip?qDPrxwjxHRHPPjp3CKWF;MDU-KHPsoJ$:-QE3ZWNHv Tgq$NuacEoGCwaR3M-:OY,zp!mB'xFGZmVJVC

After 100 steps:

CODE

1ROMEO: atit co yel soryN wil tt nole f mouu myan touthomthe thihXr d hathit t itrths mbe:
2TThef chan hedoug achaelin't mit lsen te win id ke yo tacuinere?

After 300 steps:

CODE

1ROMEO: butie tey pasthus wim trarim,
2Achent ly hatout thond thig we he, nougre, wout mecher eche ime he.
3Th aghe pleat mer hore thethe hakar y,
4Thawbeat he

After 1,000 steps:

CODE

1ROMEO:
2As wart you quest will young chansts it have.

After 3,000 steps:

CODE

1ROMEO:
2As yet away to your farrence be nothing hath.

12. The grown model and the exhibit

CODE

1exam score, grown here:       1.66
2exam score, the exhibit:      1.69
3
4grown here:
5ROMEO:
6Then the heart me, of my lovel. We grates your man
7Sommiss of the liebout of this list this more!
8
9First Consirst of the my quakes of this been queen:
10Yet death some for him your time by and girl.
11
12WA
13
14the exhibit:
15ROMEO:
16Then this of hearts fear of Contic it thy slaught?
17
18PEY:
19Come, what I will one that that that be nother,
20As then, my see own banish, thou banish'd risgue,
21And have for a horsure as in by all death.
22Wh

The grown model is an ordinary readable model, so every function in the readable model works on it. On the exam as a whole, it does as well as the exhibit, slightly better in fact, perhaps because it had no dropout. Its numbers, though, are completely different: it started from different random numbers, practised on different snippets, and had no dropout. Table by table, its weights and the exhibit's have a correlation of about 0.002, which is to say none.

And on a single question, the two can disagree a lot. The exhibit gives d after goo a 96.6% chance; the grown model gives it 32.4%, with m and k close behind. The text itself is clear: in Tiny Shakespeare, goo is followed by d 618 times out of 626.

To see whether that was a fluke, I grew a second model with the same program, changing only STARTING_NUMBERS_SEED from 42 to 7:

Model	Exam score	Chance of `d` after `goo`
Part 2's exhibit, grown with dropout	1.69	96.6%
grown here, from random start 42	1.66	32.4%
grown here, from random start 7	1.64	80.1%

The three exam scores are within 0.05 of each other, and the three answers to one question range from 32% to 97%. The two models grown here share nothing either: their weights have correlations between −0.02 and 0.03. An average over 111,540 exam questions can hide big differences on single ones. I do not know why each model ends up as sure or unsure as it does about goo, and I would not trust any story I made up to explain it.

What I learned from rewriting it

Bullet Points · What the rewrite showed me

Training really does not need to understand the model. It needs the shapes of the tables, a guess for every letter, and arithmetic that PyTorch can follow backwards. Everything that makes the model a GPT is inside a box that training barely opens.
The surprise score is one line of arithmetic. -log(chance of the right letter), averaged. Everything the model learns, it learns from that one number.
The nudge rule is two running averages and a division. AdamW sounds advanced, but written out it is a dozen named lines, and it matches PyTorch's own to within rounding.
The part I cannot rewrite readably is backward. It works, and I tested it by hand, but it is a second black box: PyTorch's record of every calculation, played in reverse.
Different numbers, the same exam score. Three models, with numbers that have nothing in common, score within 0.05 of each other. Whatever the model has learned, it is not stored in any one particular set of numbers, which is one more reason that reading meaning into single numbers is hard.
An average hides a lot. Almost the same exam score, and yet 96.6%, 32.4% and 80.1% on goo. One score cannot tell you everything a model has learned, or failed to learn.

References

MiniGPT, rewritten to be read: the black box this program uses.
MiniGPT (Part 2): growing the exhibit, and how the slopes are found.
Jibin Joseph, MiniGPT: Rebuilding GPT from First Principles, and its notebook.
Rumelhart, Hinton and Williams, Learning representations by back-propagating errors, Nature, 1986.
Kingma and Ba, Adam: A Method for Stochastic Optimization, 2014.
Loshchilov and Hutter, Decoupled Weight Decay Regularization (AdamW), 2017.
Zhang and others, Why are Adaptive Methods Good for Attention Models?, NeurIPS 2020.
Kunstner and others, Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be, ICLR 2023.
Andrej Karpathy, Tiny Shakespeare, the text the model learns from.