LLM Benchmarking

Neil Haddley • November 12, 2026

A from-scratch version of my accounting benchmark: 30 questions that know their own answers, a checker instead of a judge, and six models run through Ollama

AIPythonRunning models locally with Ollamabenchmarkingevaluationollamaaccountingqwengemma

I already have a benchmarking tool for accounting questions. llm-accounting-benchmark is a Streamlit app. It holds 285 prompts, sends them to local and cloud models, asks a second model to mark every answer against a rubric (accuracy, completeness and clarity, ten points each), and keeps the results in SQLite. It works. It is also a lot of machinery, and the idea gets lost in it.

So I started again from nothing, with one rule: build the smallest benchmark that still teaches me something true. This post walks through it. It is one folder with a handful of small files, no database and no second model to do the marking. I ran it for real on six models.

The five parts of any benchmark

Big or small, every benchmark is the same five boxes. Watch the question travel through them.

One real question walks through the five parts. Its answer, 7,790.00, goes straight to the checker, and the model never sees it. The model puts the signs the wrong way round and says 8,290.00, the checker catches it, and the red square on the scoreboard is that question.Scroll sideways, or tap the diagram to open it full size

One real question walks through the five parts. Its answer, 7,790.00, goes straight to the checker, and the model never sees it. The model puts the signs the wrong way round and says 8,290.00, the checker catches it, and the red square on the scoreboard is that question.

  1. Questions with the right answer attached. Without the right answer there is nothing to mark against.
  2. A prompt. The question plus the rules for replying. The rules matter more than you would think, as the last section of results shows.
  3. A model. Whatever answers. Here it is a model running on my own machine through Ollama.
  4. A checker. Reads the reply and decides whether it is right.
  5. A scoreboard. Adds up the marks, and the time each answer took.

The big project and this one have all five. They differ in boxes 1 and 4, and that difference is the whole post.

Questions that know their own answers

The big project's prompts are almost all multiple choice, so a letter checker can mark nearly all of them. A judge is still needed for the three open-ended prompts and for the quality of the explanation, because a model can answer in any shape and explaining takes judgement.

I took the opposite route. questions.py builds 30 questions with code, and the same code works out the right answer. There are six topics, picked to echo areas the big project covers (bank reconciliation, depreciation, GST, margins, financial statements and inventory), and five levels in each. Level 1 needs one step. Level 5 needs five.

Here is the same question at its easiest and its hardest. Only the second line changes the arithmetic, but notice how much more the model has to keep track of.

PROMPT
1Level 1: The cash book shows a balance of $5,150.00. Bank charges of $80.00 are on the bank statement but not yet in the cash book. What balance appears on the bank statement?
PROMPT
1Level 5: The cash book shows a balance of $3,930.00. Bank charges of $60.00 are on the bank statement but not yet in the cash book. Interest of $25.00 was credited by the bank but is not yet in the cash book. Cheques totalling $350.00 were issued but have not yet been presented to the bank. A deposit of $360.00 was made but has not yet been credited by the bank. A customer's cheque for $330.00 was dishonoured; the bank has deducted it but the cash book still shows it. What balance appears on the bank statement?

The sum that produces the right answer is five lines of Python, kept apart from the question text so that it can be tested:

PYTHON
1def bank_statement_balance(cash_book, charges=0.0, interest=0.0,
2                           unpresented=0.0, uncredited=0.0, dishonoured=0.0):
3    """Balance on the bank statement, from the cash book balance and the reconciling items."""
4    return round(cash_book - charges + interest + unpresented - uncredited - dishonoured, 2)
Under the hood: Why the questions never change

Every question comes from a random number generator that starts from one fixed seed (20261112). Run the program tomorrow, or on your machine, and you get the same 30 questions with the same numbers. That is what lets two people compare results: they asked the same thing. The sums are covered by 10 small tests in test_questions.py, which check them against answers I worked out by hand.

Asking, then marking

Each question goes to the model with the same instructions. Here they are in full:

PROMPT
1You are helping with an accounting exam. Work out the answer step by step, keeping it short, then finish with one final line in exactly this form:
2ANSWER: <number>
3Use digits only: no currency symbol, no commas and no units. Use a minus sign for a loss or a refund. Round to 2 decimal places if needed.
4
5Question:
6...

The final line is the trick that makes marking easy. minibench.py looks for the last ANSWER: line, reads the number, and compares it with the right answer to within one cent. If the line is missing, the answer is marked wrong, and the report counts those separately so that I can tell "got the sum wrong" from "did not follow the format".

The settings are chosen to make the run repeatable. The temperature is 0 and the random seed is fixed, so a model should give the same answer to the same question. The model's own "thinking" mode is switched off, so that every model is judged on what it writes in the reply. Answers are cut off at 700 tokens.

Two ways to mark the same answer. The checker only works when the question has one right number.Scroll sideways, or tap the diagram to open it full size

Two ways to mark the same answer. The checker only works when the question has one right number.

The runs

I ran the same 30 questions on six models. Every model is the build Ollama downloads by default, which is a compressed (quantised) copy of the original. The table shows which compression each one uses, because they are not all the same.

ModelParametersCompressionTime for the 30 questionsSpeed (tokens a second)
Gemma 4 26B25.8B (mixture of experts)Q4_K_M5.3 minutes17.5
Qwen3.5 9B9.7Bnvfp44.2 minutes34.6
Gemma 4 12B11.9BQ4_K_M11.1 minutes9.3
Qwen3.5 2B2.3BQ8_03.3 minutes38.0
Qwen2.5 14B14.8BQ4_K_M13.0 minutes7.1
Llama 3.1 8B8.0BQ4_K_M4.3 minutes18.6

The times add up the seconds each answer took, without the one-off wait while the model loads into memory.

The scoreboard

All six models answered the same 30 questions, with the working shown. The bars grow as the answers are counted.

Share of the 30 questions answered correctly. The numbers in brackets are right answers out of 30.Scroll sideways, or tap the diagram to open it full size

Share of the 30 questions answered correctly. The numbers in brackets are right answers out of 30.

Gemma 4 26B got 29 of 30 right. Qwen3.5 9B got 27, Gemma 4 12B got 26, Qwen3.5 2B got 23, Qwen2.5 14B got 21, and Llama 3.1 8B got 15, which is half.

Look at the middle of the table. Qwen2.5 14B is more than six times the size of Qwen3.5 2B, and it scored lower: 21 against 23. A bigger number on the label is not a better model.

Every answer, one square each

The scoreboard hides where the mistakes are. This picture does not. Each square is one question: green is right, red is a wrong number, and amber means the model never finished with an ANSWER: line. The squares light up question by question.

Six models down the side. Six topics across the top, with the questions getting harder, 1 to 5 steps, inside each topic.Scroll sideways, or tap the diagram to open it full size

Six models down the side. Six topics across the top, with the questions getting harder, 1 to 5 steps, inside each topic.

Two things jump out. The first is the reconciliation column, which is red almost everywhere: across all six models only 12 of the 30 reconciliation answers were right, compared with 29 of 30 for the financial statements questions, which are plain arithmetic with named parts. Llama 3.1 8B got none of the five reconciliation questions right.

The second is that 13 of the 30 questions were answered correctly by every model. Those questions tell me little about which model is better. A bigger benchmark would spend its questions on the space between "everybody gets it" and "nobody gets it".

The question every model missed

One question, reconciliation-4, was missed by all six models, and one, reconciliation-5, was solved only by Gemma 4 26B. My own rule applies: when everyone fails, suspect the question first.

So the benchmark did find a real weakness, and I would not have believed it without doing the sum myself.

One more detail: Qwen3.5 9B was the model that gave no final line. It wrote on until it reached the 700-token limit on this question, so it was marked wrong, and the report counts it as a missing answer, not a wrong number. Across the 180 answers where the models showed their working, that was the only time a model failed to give an ANSWER: line.

More steps, more mistakes?

Each topic has five levels, and each level needs one more step than the one before. Here is how the models did at each level. The lines draw themselves from left to right.

Share of questions right at each level. Every level has six questions, one from each topic.Scroll sideways, or tap the diagram to open it full size

Share of questions right at each level. Every level has six questions, one from each topic.

Across all six models, level 1 had 35 of 36 answers right, level 2 had 30, level 3 had 27, level 4 had 23 and level 5 had 26. So, mostly yes: more steps meant more mistakes. The line is not smooth, and it should not be: six questions per level is thin, and level 4 contains the reconciliation question that everyone missed. I would not draw a trend line through five points.

Fast, or right?

Accuracy is only one axis. Here is how quickly each model wrote its answers, against how many it got right.

Each dot is a model. Further right is faster (tokens written per second); higher is more accurate.Scroll sideways, or tap the diagram to open it full size

Each dot is a model. Further right is faster (tokens written per second); higher is more accurate.

The best spot is the top right, and the Qwen3.5 9B is nearest to it: 27 right at about 35 tokens a second. Qwen3.5 2B is the fastest at 38.0 tokens a second, and still got 23 right. Gemma 4 12B and Qwen2.5 14B are slow, at 9.3 and 7.1 tokens a second: an answer took them 22.3 and 26.0 seconds on average, against 6.6 for the 2B.

The surprise is Gemma 4 26B. It is more than twice the size of the 12B, yet it wrote at 17.5 tokens a second, nearly twice as fast, and answered in 10.7 seconds on average instead of 22.3.

Under the hood: Why a bigger model can be faster

Ollama can show a model's metadata. Gemma 4 26B lists 128 experts, with 8 used for each token. That is a mixture-of-experts design: the model holds a lot of knowledge, but for each word it writes, only a few of its experts do any work. Its speed looks more like a small model's than a 26-billion-parameter one's, while its knowledge comes from the whole. The Gemma 4 12B has no experts listed, so every part of it works on every token. I checked the metadata, not the design papers, so treat this as the most likely reason rather than a proven one.

A caution on this picture. The Qwen3.5 9B uses a different compression (nvfp4 against the Q4_K_M that most of the others use), so its position on the speed axis is a guide, not a measurement I would compare to the decimal.

The same model, told two ways

So far every model was asked to show its working. What happens if I ask for the answer and nothing else? I ran three of the models a second time with a different instruction, and nothing else changed:

PROMPT
1You are helping with an accounting exam. Do not show any working. Reply with one line only, in exactly this form:
2ANSWER: <number>
The same questions, the same models, the same checker. Only the instruction changed.Scroll sideways, or tap the diagram to open it full size

The same questions, the same models, the same checker. Only the instruction changed.

Qwen3.5 2B fell from 23 right to 1. Qwen3.5 9B fell from 27 to 8, and Gemma 4 12B fell from 26 to 8. The answers came back quickly, and they were mostly wrong. With working, the models wrote between 179 and 269 tokens an answer. Without it they wrote 10 to 12 tokens, which is just the number.

My explanation, which I did not test, is that a language model does its arithmetic by writing. The steps are the computation. Take away the room to write them, and it has to produce the final number in one go.

Try it yourself

What a benchmark is for, and what it is not

A benchmark answers one narrow question: how good is this model at solving this kind of problem using only what is stored in its weights? That is a useful thing to know. If I give a model an accountancy exam question, the answer comes from what it absorbed in training and nothing else. A model that understands standard accounting terms should be more useful to an accountant than one that gets confused by them, and a benchmark is how I check that instead of guessing.

But a real problem is not an exam question, and a real answer needs more than a good model. It needs a harness: everything around the model that decides what it is asked, what it can look at, what it can use, and what happens to its answer before anyone relies on it.

Here is what a harness adds, with each item tied to something I saw in this post:

What the harness providesWhat I saw in this benchmark
The right instructionThe same three models scored 1, 8 and 8 when I asked for the answer only, against 23, 27 and 26 when I let them show their working. The prompt is part of the harness, and it moved the score more than the choice of model.
A calculatorThe models do their arithmetic by writing it out. A harness that hands the sums to code removes a whole class of slips, and the model only has to decide which sums to ask for.
A check on the answerAll six models gave a wrong answer to reconciliation-4, and the same sign mistake was in more than one of them. The checker found it because my program knew the answer. In real work nobody knows the answer in advance, so a harness needs a second route to the same number, a rule, or a person who signs off.
A way to cope with a missing answerQwen3.5 9B once wrote on until it ran out of room and never gave a final line. A harness notices that, and asks again or sends it to a person, instead of passing on nothing.
The facts of the caseAn exam question contains everything needed. A real client question needs the ledger, the invoice, the tax year and the rules in force. The model has none of those stored, so a harness has to fetch them and put them in front of it.
A recordI saved every raw answer so I could mark them again after I found my own checker bug. A harness keeps that record so any answer can be traced, repeated and corrected later.

I have not built a harness for these questions, so the table is reasoning from what these runs showed, not a test of it. A good follow-up is to run the same 30 questions through a loop that gives the model a calculator and a second chance, and see which of the lost points come back.

What this does not tell you

  • It is small. Thirty questions is enough to see big differences, such as a model that gets most right and one that gets half. It is not enough to rank two models that are three questions apart.
  • Each question was asked once. At temperature 0 a model should give almost the same answer each time, so repeating the run would tell me less about luck than it would for a sampled model. It also means I have no measure of how a model behaves when you let it vary.
  • It only marks sums. A model could write brilliant explanations and still fail here, and the reverse. The judge in the big project exists for the questions that have no single number.
  • The questions are made up by me. They are the shape of real bookkeeping problems, not real ones. A model might have seen a textbook version of a bank reconciliation.
  • The numbers belong to these exact files. A different seed, a different prompt or a different quantisation would give different scores. That is why the files, the seed and the settings are all in the download.

Download

WhatFile
The 30 questions, their sums and the checkerquestions.py
Ask a model and save the answersminibench.py
Print the scoreboardreport.py
Tests for the sums and the checkertest_questions.py
The runs behind this postrun_all.sh
Every answer from every runresults.json