LLM Benchmarking, Part 2: Fifteen Models, 282 Questions
Neil Haddley • November 13, 2026
I ran 282 accounting multiple-choice questions through fifteen models, first answering at once with a single letter, then with brief reasoning, had a judge mark the working, and measured which questions every model gets right.
In LLM Benchmarking I wrote 30 questions that know their own answers, so that a few lines of code could mark them. This post runs the same idea on the 282 multiple-choice accounting questions from my older llm-accounting-benchmark.
The questions
The 282 questions are the MMLU professional accounting test (cais/mmlu, subject professional_accounting, test split). Each has four options and one right answer, so a letter checker can mark every one.
What I ran
Two passes, so that I could separate knowing from thinking:
- One letter. The prompt ends with "Reply with only the letter of the correct option". The model gets no room to work anything out.
- Brief reasoning. The prompt ends with "Think it through briefly, then end your reply with a final line in exactly this form: ANSWER: X". The model can reason, and the checker reads the last
ANSWER:line.
In part 1 I asked every model to finish with a line in a fixed form, such as "ANSWER: B". The checker only has to find that line and read the letter after it, so it does not have to guess which option a long answer is pointing at.
The runner builds each prompt from the question, its four options and one of those two instructions, and sends it to Ollama with the temperature at 0. This is the core of run.py, shortened: the real file also resumes a stopped run and has a DeepSeek branch.
PYTHON
1INSTR = { 2 "direct": "Reply with only the letter of the correct option (A, B, C or D) and nothing else.", 3 "reason": "Think it through briefly, then end your reply with a final line in exactly this form: ANSWER: X (where X is A, B, C or D).", 4}[mode] 5 6for i, o in enumerate(orig): 7 opts = "\n".join(f"{'ABCD'[k]}. {c}" for k, c in enumerate(o["choices"])) 8 prompt = f"{o['question']}\n\nOptions:\n{opts}\n\n{INSTR}" 9 body = {"model": model, "messages": [{"role": "user", "content": prompt}], "stream": False, 10 "options": {"temperature": 0, "seed": 1, "num_predict": 12 if mode == "direct" else NUM_PREDICT}} 11 reply = json.load(urllib.request.urlopen(request(body))) 12 record = {"i": i, "key": "ABCD"[o["answer"]], "text": reply["message"]["content"]}
The "direct" pass caps the answer at 12 tokens, and the "reason" pass at the cap I set for that model.
I ran fifteen models at temperature 0. Thirteen ran locally through Ollama on my Mac, from a 3B Llama to the 72B Qwen2.5 and the 70B Llama 3.3. Two ran through the DeepSeek API: deepseek-flash and deepseek-v4-pro. For those two I turned thinking off in the one-letter pass.
The checker is grade.py, shown here without the scoring table. It takes the last ANSWER: line, and only if there is none does it fall back to a longer reader, reader.py, that looks for phrases such as "the correct answer is":
PYTHON
1def letter(text, mode): 2 m = re.findall(r"ANSWER:\s*\**\s*\(?([A-D])\b", text, re.I) 3 if m: 4 return m[-1].upper() 5 if mode == "direct": 6 m = re.match(r"^\W*([A-D])\b", text.strip(), re.I) 7 if m: 8 return m.group(1).upper() 9 return read(text) # None when no letter can be found 10 11right = sum(letter(r["text"], mode) == r["key"] for r in records)
An answer with no clear letter counts as wrong. Thinking models can spend their whole allowance thinking and never reach the last line, so the cap on the answer length matters. I gave the Qwen3 and Qwen3.8 models 4,000 tokens, deepseek-r1 and qwq 6,000 tokens, and the rest 1,500. Models that were cut off are in the "no letter" column.
The results
| Model | Reasoning: right of 282 | No letter | One letter: right of 282 | Reasoning time |
|---|---|---|---|---|
| deepseek-flash (API) | 97.9% | 0 | 84.0% | 15 min |
| deepseek-v4-pro (API) | 97.5% | 5 | 80.1% | 41 min |
| Qwen3.8 27B (MLX) | 93.6% | 1 | 73.0% | 129 min |
| QwQ 32B | 86.5% | 16 | no answer | 792 min |
| Qwen3 14B | 85.1% | 8 | 61.7% | 165 min |
| Qwen2.5 72B | 85.1% | 1 | 70.9% | 285 min |
| Llama 3.3 70B | 84.8% | 2 | 69.9% | 565 min |
| DeepSeek-R1 32B | 83.3% | 11 | no answer | 479 min |
| Qwen3 8B | 80.1% | 19 | 52.5% | 134 min |
| Qwen2.5 14B | 79.4% | 0 | 63.1% | 53 min |
| Gemma 3 27B | 78.4% | 1 | 62.4% | 103 min |
| gpt-oss 20B | 78.0% | 25 | no answer | 38 min |
| Gemma 3 12B | 63.5% | 1 | 52.8% | 43 min |
| Llama 3.1 8B | 51.4% | 23 | 47.9% | 34 min |
| Llama 3.2 3B | 46.8% | 19 | 41.1% | 14 min |
(Chance is 25%. In the one-letter pass, deepseek-r1, qwq and gpt-oss returned an empty reply to all 282 questions: they are thinking models, and the 12 tokens I allowed were spent thinking before any letter appeared. That pass tells me nothing about them, so the table says "no answer" and not 0%.)
Four things stand out.
Among the models I ran locally, more parameters was not the main thing. (A parameter is one of the numbers a model learned in training. "72B" means 72 billion of them.) The 72B Qwen2.5 and the 14B Qwen3 both score 85.1%. The 70B Llama 3.3 scores 84.8%, and the 27B Qwen3.8 beats all three by 8 points or more. A newer family at 14B matched an older family at 72B, and used about a fifth of the memory. The two hosted DeepSeek models are a different case. I did not look up their sizes, but they are very likely far larger than anything I ran locally, and they scored best, so I cannot say size played no part there. They also differ from the local models in other ways, such as how they were trained, so this test cannot separate size from everything else.
Letting the model think was worth 10 to 25 points. For the models that can do both, the reasoning pass beat the one-letter pass by 11 points for Gemma 3 12B, 14 for Qwen2.5 72B, 16 for Qwen2.5 14B, 21 for Qwen3.8 27B and 23 for Qwen3 14B. The small Llama models gained only 4 to 6. The one-letter numbers also show how little the 70B models know without thinking: both land near 70%.
Thinking costs time, and more thinking is not always better. QwQ 32B spent about 13 hours and 1,800 tokens an answer to reach 86.5%. Qwen3.8 27B took 2 hours and 800 tokens an answer to reach 93.6%. The long thinkers also hit my caps: gpt-oss 20B, Qwen3 8B and QwQ lost 25, 19 and 16 answers to "no letter", so their scores are a little lower than they would be with a larger cap.
The hosted models were far ahead, and the "flash" one won. deepseek-flash finished all 282 questions in 15 minutes and deepseek-v4-pro in 41, and the "flash" model scored 0.4 points higher. They are 1 question apart, so I would not read that as flash being better. Flash is also cheaper. DeepSeek's price list at the time of writing (api-docs.deepseek.com) charges $0.30 per million input tokens and $1.20 per million output tokens at peak times for flash, and $1.32 and $3.96 for pro, which is 3 to 4 times as much. Pro also wrote about twice as many tokens per answer (782 against 409). By my estimate, the reasoning pass over all 282 questions cost about 15 cents with flash and about a dollar with pro at peak prices. I worked that out from the token counts and did not read a bill.
Marking the thinking
A letter says whether a model landed on the key. It does not say whether the working was any good. I added a judge to mark that. The judge is gemma4:12b, running locally. It is not one of the 15 models, so it never marks its own answers. It does share a family name with the two Gemma 3 models it marks, and I cannot rule out that it favours them. For every reply it sees the question, the four options, the correct answer and the model's reply, and it gives three scores from 1 to 10 (Accuracy, Completeness and Clarity) and a verdict: SOUND, SLIP (a small error), FLAWED or UNSUPPORTED (a claim with no reasoning behind it).
The judge can only mark what a model wrote in its reply. Qwen3 14B, deepseek-flash and gpt-oss 20B do their thinking in a hidden field and reply with little more than "ANSWER: X", and the judge gave Qwen3 14B a Completeness of 1 on nearly every answer. I left those models out of this section. The seven below write their working in the reply.
| Model | Right (checker) | Accuracy | Completeness | Clarity | SOUND | FLAWED |
|---|---|---|---|---|---|---|
| Qwen3.8 27B | 93.6% | 9.6 | 9.9 | 9.9 | 264 | 17 |
| Qwen2.5 72B | 85.1% | 9.0 | 9.5 | 9.8 | 238 | 42 |
| Llama 3.3 70B | 84.8% | 8.9 | 9.4 | 9.7 | 235 | 43 |
| Qwen2.5 14B | 79.4% | 8.5 | 9.4 | 9.8 | 217 | 62 |
| Gemma 3 27B | 78.4% | 8.4 | 9.4 | 9.8 | 218 | 62 |
| Gemma 3 12B | 63.5% | 7.4 | 8.9 | 9.6 | 176 | 99 |
| Llama 3.1 8B | 51.4% | 6.2 | 7.6 | 8.8 | 140 | 140 |
(Scores are means over 282 answers. A few verdicts are missing where the judge left the field out, so the SOUND and FLAWED counts can add to a little less than 282.)
The radar puts the checker beside the three judge scores, multiplied by ten so that all four axes run from 0 to 100.
Clarity and Completeness sit near the rim for every model except the weakest: all seven write clear, thorough-looking explanations. The two axes that depend on the key, Right answer and Accuracy, separate the models. The judge holds the key, so its Accuracy follows the checker closely, and I would not count it as a second opinion on the answer. What it adds is the verdict on the working, and it found little: of 1,974 answers, only 16 were right with FLAWED thinking (a lucky guess), and only 2 were wrong with SOUND thinking.
I also asked deepseek-v4-pro to judge 60 of the same answers, picked at random. The two judges put 58 of the 59 comparable answers in the same group, sound (SOUND or SLIP) or flawed. DeepSeek scored Completeness and Clarity about 0.6 points lower than Gemma. The verdicts are steady across two judges from different families. The finer scores are a matter of taste.
Do the models get the same questions right?
I wanted to know whether the models share their hard questions, or whether each one fails on its own. I took the 15 reasoning runs, which all have 282 answers, and counted for each question how many models got it right.
| Models right | 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 | 7 or fewer |
|---|---|---|---|---|---|---|---|---|---|
| Questions | 61 | 60 | 37 | 31 | 20 | 17 | 11 | 12 | 33 |
(The seven-or-fewer bucket is 4 + 13 + 5 + 7 + 1 + 2 + 1 + 0 questions, for 7, 6, 5, 4, 3, 2, 1 and 0 models.)
61 questions (22%) were answered correctly by all 15 models, and no question was missed by all 15.
The interesting part is who fails together. I measured how alike two models' right-and-wrong patterns are with phi: 1 means identical patterns and 0 means unrelated. Pairs from the same family score higher than pairs from different families. The 26 same-family pairs average 0.35 and the 79 different-family pairs average 0.25. (I counted DeepSeek-R1 32B with the Qwen models, because it is a distilled Qwen.) The most alike pair is Qwen2.5 14B and Qwen2.5 72B, at 0.50. The least alike is deepseek-flash against the 3B Llama, at 0.04. A model from the same family as one that is wrong is more likely to be wrong on the same question, and that is why a second opinion from another family is worth more than one from the same family.
Under the hood: What phi is
Phi answers one question: when one model gets a question wrong, is the other model more likely to get it wrong too? It is the ordinary correlation, applied to a list of right and wrong marks.
For two models I sort the 282 questions into four boxes:
| Qwen2.5 14B and Qwen2.5 72B | 72B right | 72B wrong |
|---|---|---|
| 14B right | 211 | 13 |
| 14B wrong | 29 | 29 |
Both models are right most of the time, so they would agree often even if they were unrelated. Phi removes that effect by comparing the diagonal (both right, both wrong) with the other two boxes, and scaling the result:
phi = (211 × 29 − 13 × 29) ÷ √(224 × 58 × 240 × 42) = 5,742 ÷ 11,444 = 0.50
- 1 means the two models are right and wrong on exactly the same questions.
- 0 means that knowing one model's mark tells you nothing about the other's.
- Below 0 would mean one tends to be right where the other is wrong. No pair here goes below 0.
Compare deepseek-flash with the 3B Llama. Llama is wrong on 150 of the 282 questions, and DeepSeek is wrong on only 6. If DeepSeek's mistakes had nothing to do with Llama's, I would expect about 3 of those 6 to be among Llama's 150, and 4 are. Phi is 0.04.
The counting is a few lines of overlap.py, shortened here. It marks every answer right or wrong with the same letter() function, adds up the models that got each question right, and measures how alike two models are with phi:
PYTHON
1right = {} # model -> {question number: True or False} 2for path in glob.glob("runs/*__reason.jsonl"): 3 records = [json.loads(line) for line in open(path)] 4 if len(records) == 282: # only models that answered every question 5 right[model_of(path)] = {r["i"]: letter(r["text"], "reason") == r["key"] for r in records} 6 7count = [sum(right[m][i] for m in right) for i in range(282)] # models right, per question 8print(collections.Counter(count)) # how many questions had 15, 14, 13 ... models right 9 10def phi(a, b): 11 n11 = sum(right[a][i] and right[b][i] for i in range(282)) # both right 12 n10 = sum(right[a][i] and not right[b][i] for i in range(282)) # only a right 13 n01 = sum(right[b][i] and not right[a][i] for i in range(282)) # only b right 14 n00 = 282 - n11 - n10 - n01 # both wrong 15 d = math.sqrt((n11 + n10) * (n01 + n00) * (n11 + n01) * (n10 + n00)) 16 return (n11 * n00 - n10 * n01) / d
The questions that most models miss are interesting. Four stand out. Three were answered correctly by three or fewer models, and the fourth by only four:
- A question about audit engagements. "Which condition would most likely raise questions about the integrity of management?" The key says B, that the CPA is not permitted to see the senior managers' salaries. Fourteen of the 15 models chose C, substantial inventory write-offs just before each year end. Only
deepseek-flashagreed with the key. I think C is a defensible answer, and I would not call the 14 wrong without seeing the source. - A question about compilations. "Which factor would most likely cause an accountant not to accept an engagement to compile the financial statements of a nonissuer?" The key says B, that management does not investigate reports of asset misappropriation. Thirteen models chose C, that the entity plans to leave out nearly all the disclosures GAAP requires. Only the two DeepSeek models matched the key.
- A bond question about straight-line against effective interest amortisation, where six models never reached a letter and six chose A. Only
deepseek-flashand Qwen3.8 27B got it right. - A question about comparing investment projects. The question is "When comparing investment projects:" with four options. The key says C, "The IRR never changes the ranking of the projects". IRR is the internal rate of return, and it can rank projects differently from NPV, so I do not see how C is true. Only 4 of the 15 models chose C. I think the key is wrong here, but I have not checked the source.
The two DeepSeek models miss different questions. deepseek-flash missed 6 and deepseek-v4-pro missed 7, and only 2 questions appear on both lists. Between them they got 271 questions right together, and missed 2 together.
Who gets the hard questions
A total score hides where a model loses its points. I sorted the questions by how many of the 15 models got them right, and then asked each model what share of each group it got right. The 61 questions that all 15 models got right are left out, because every model scores 100 on them. Each axis below is a group of questions, labelled by how many of the 15 models got them right, and a model that is strong all round fills the whole pentagon.
All four get nearly every question that 13 or more models get right. They separate on the two hardest groups. deepseek-flash gets 88% of the 33 questions that 7 or fewer models got right. Qwen3.8 27B gets 73%. Qwen3 14B gets 15% and Qwen2.5 72B 27%, so most of the gap between them and the top two is in those 33 questions.
The two small Llama models fail even the easier groups: the 3B gets 53% of the questions that 14 models got right. From 14B up the shapes are alike: Qwen3 14B is the same shape as Qwen2.5 72B, and it does better on the middle groups. More parameters did not extend the reach into the hardest group, where the 14B and the 72B models both fall below 30%.
(The groups are made from the same 15 models, so a model that is wrong on a question helps to put it in a harder group. Read the charts as a description of these 15 models, not as a measure of difficulty that would hold for others.)
Why the hard questions are hard
Ten of the eleven questions that 4 or fewer models got right ask for a specific rule or standard: auditing, compilation engagements, contract law, tax and government reporting. Only the investment question is about finance in general. None needs a long calculation. I read the replies to the two hardest, and the wrong ones share a pattern. They choose the option that looks most suspicious to a general reader, and the right answer needs the professional standard.
- On the audit question, Qwen3.8 27B dismisses the key's answer B as "a scope limitation issue" that "doesn't directly implicate management's integrity", and calls C, inventory write-offs before year end four years running, "a major red flag".
deepseek-flash, the only model right, reasons from the audit rule: refusing the CPA access to information raises doubts about management's honesty. - On the compilation question, Qwen3.8 27B treats B as "concerning but not necessarily disqualifying", because a compilation does not look for fraud.
deepseek-flashstates the rule: omitting nearly all GAAP disclosures is generally allowed in a compilation if the report says so, so C is not a reason to decline.
I also asked the judge to explain, for each of the 97 wrong replies on those eleven questions, where the reasoning went wrong, and to label the mistake. I did not use its labels. It called 76 of the 97 "wrong rule", which says more about the judge's habit than about the models. Some of its explanations are also built to fit the key: on the compilation question it told one model that refusing to investigate misappropriation was an integrity risk that should have ended the engagement, which restates the key without testing it.
What I take from this
- Among the local models, generation and the room to think mattered more than the number of parameters. A 14B model from a newer family matched a 72B model from an older one, and letting every model reason added 10 to 25 points.
- Models fail together. 22% of the questions were easy for all fifteen, and none were impossible for all fifteen. A second opinion from a different family is worth more than a second run from the same one.
- A judge with the key checks conclusions, not working. It found 16 lucky guesses in 1,974 answers and missed 39 of 40 planted number errors.
- The answer cap shapes the result. Models that think can hit the cap and lose the answer, so the bottom of the table is a little pessimistic for them.
What comes next
I would like to judge the thinking models too, by saving their hidden thinking and not only their replies. I would also like to use the open-ended bank reconciliation prompts in the older project, which a letter checker cannot mark, and to try a judge that does not see the key.
The code
These are the files in full. To run one model: python3 run.py qwen3:14b reason, then python3 grade.py to score every answer file, python3 overlap.py to count how many models get each question right and python3 judge.py gemma4:12b qwen2.5_72b to judge one model's answers.
run.py asks a model all 282 questions and saves its answers.
PYTHON
1"""Run MMLU professional accounting (282 questions) through Ollama. usage: run.py MODEL MODE [limit] 2MODE direct: reply with one letter. MODE reason: brief reasoning then 'ANSWER: X'.""" 3import json,sys,time,re,urllib.request,os 4model,mode=sys.argv[1],sys.argv[2]; limit=int(sys.argv[3]) if len(sys.argv)>3 else 0 5orig=json.load(open('orig.json')) 6out=f"runs/{model.replace('/','_').replace(':','_')}__{mode}.jsonl" 7os.makedirs('runs',exist_ok=True) 8done={json.loads(l)['i'] for l in open(out)} if os.path.exists(out) else set() 9INSTR={'direct':"Reply with only the letter of the correct option (A, B, C or D) and nothing else.", 10 'reason':"Think it through briefly, then end your reply with a final line in exactly this form: ANSWER: X (where X is A, B, C or D)."}[mode] 11def deepseek_key(): 12 k=os.environ.get("DEEPSEEK_API_KEY") 13 if k: return k.strip() 14 p=os.path.expanduser("~/Documents/GitHub/llm-accounting-benchmark/.env") 15 if os.path.exists(p): 16 for l in open(p): 17 if l.startswith("DEEPSEEK_API_KEY="): return l.split("=",1)[1].strip().strip('"').strip("'") 18 return None 19def call_deepseek(prompt,num_predict): 20 key=deepseek_key() 21 if not key: sys.exit("DEEPSEEK_API_KEY not set (environment or llm-accounting-benchmark/.env)") 22 name=model.split(":",1)[1] 23 body={"model":name,"messages":[{"role":"user","content":prompt}],"max_tokens":num_predict} 24 body["temperature"]=0 25 if mode=="direct": body["thinking"]={"type":"disabled"} 26 req=urllib.request.Request("https://api.deepseek.com/chat/completions",json.dumps(body).encode(),{"Content-Type":"application/json","Authorization":"Bearer "+key}) 27 r=json.load(urllib.request.urlopen(req,timeout=1800)) 28 return {"message":{"content":r["choices"][0]["message"].get("content") or "","thinking":r["choices"][0]["message"].get("reasoning_content") or ""},"eval_count":r.get("usage",{}).get("completion_tokens")} 29def call(prompt,num_predict,think): 30 if model.startswith("deepseek:"): 31 return call_deepseek(prompt,num_predict if mode=="direct" else 8000) 32 body={"model":model,"messages":[{"role":"user","content":prompt}],"stream":False,"options":{"temperature":0,"seed":1,"num_predict":num_predict}} 33 if think is not None: body["think"]=think 34 req=urllib.request.Request("http://localhost:11434/api/chat",json.dumps(body).encode(),{"Content-Type":"application/json"}) 35 return json.load(urllib.request.urlopen(req,timeout=1800)) 36n=0 37with open(out,'a') as f: 38 for i,o in enumerate(orig): 39 if limit and n>=limit: break 40 n+=1 41 if i in done: continue 42 opts="\n".join(f"{'ABCD'[k]}. {c}" for k,c in enumerate(o['choices'])) 43 prompt=f"{o['question']}\n\nOptions:\n{opts}\n\n{INSTR}" 44 t=time.time() 45 try: 46 think=False if mode=='direct' else None 47 try: r=call(prompt,12 if mode=='direct' else int(os.environ.get('NUM_PREDICT','1500')),think) 48 except urllib.error.HTTPError as e: 49 r=call(prompt,12 if mode=='direct' else int(os.environ.get('NUM_PREDICT','1500')),None) 50 txt=r['message'].get('content','') 51 rec={"i":i,"key":'ABCD'[o['answer']],"text":txt,"thinking_len":len(r['message'].get('thinking','') or ''),"tokens":r.get('eval_count'),"secs":round(time.time()-t,1)} 52 except Exception as e: 53 rec={"i":i,"key":'ABCD'[o['answer']],"text":"","error":str(e)[:200]} 54 f.write(json.dumps(rec)+"\n"); f.flush() 55print('done',model,mode)
reader.py finds a letter in an answer that has no ANSWER: line.
PYTHON
1import re 2L='[A-D]' 3NEG=re.compile(r'(incorrect|wrong|not correct|not the|eliminate|rule out|ruled out|false|invalid|cannot be|can\'t be|isn\'t|is not)',re.I) 4PATS=[ 5 # strongest: explicit final statements 6 (r'(?:final answer|answer|correct (?:option|answer|choice)|correct mathematical choice|i would choose|i choose|i\'d choose|best (?:option|answer|choice)|the (?:answer|choice) is|therefore,? (?:the )?(?:answer|option|choice) is)\W{0,12}(?:is\W{0,6})?(?:option\W{0,3})?\(?\**('+L+r')\b(?!\w)',), 7 (r'\\boxed\{\s*\\?(?:text\{)?\(?('+L+r')\b',), 8 (r'\*\*\s*(?:option\s*)?('+L+r')\s*[.:)]?\s*[^\n*]{0,60}\*\*\s*$',), 9] 10def read(text): 11 t=text.strip() 12 if not t: return None 13 low=t 14 cands=[] 15 for (p,) in PATS: 16 for m in re.finditer(p,low,re.I|re.M): 17 ctx=low[max(0,m.start()-40):m.end()+30] 18 if re.search(r'(option|choice)\s*'+m.group(1)+r'\b[^.\n]{0,40}(incorrect|wrong|not)',ctx,re.I): continue 19 cands.append((m.start(),m.group(1).upper())) 20 if cands: return max(cands)[1] 21 # last three lines fallback: line that starts with letter or "Option X" 22 tail=[l for l in t.split('\n') if l.strip()][-3:] 23 for l in reversed(tail): 24 m=re.match(r'^\W*(?:option\s*)?\(?('+L+r')\)?[.:)\s]',l.strip(),re.I) 25 if m and not NEG.search(l[:40]): return m.group(1).upper() 26 m=re.search(r'(?:option|choice)\s*\(?('+L+r')\b',l,re.I) 27 if m and not NEG.search(l): return m.group(1).upper() 28 return None
grade.py scores every answer file.
PYTHON
1import json,re,sys,glob,collections 2sys.path.insert(0,'.') 3from reader import read 4def letter(t,mode): 5 m=re.findall(r'ANSWER:\s*\**\s*\(?([A-D])\b',t,re.I) 6 if m: return m[-1].upper() 7 if mode=='direct': 8 m=re.match(r'^\W*([A-D])\b',t.strip(),re.I) 9 if m: return m.group(1).upper() 10 return read(t) 11rows=[] 12for p in sorted(glob.glob('runs/*.jsonl')): 13 mode=p.split('__')[1][:-6]; model=p.split('/')[1].split('__')[0] 14 recs=[json.loads(l) for l in open(p)] 15 n=len(recs); ok=sum(1 for r in recs if letter(r['text'],mode)==r['key']); nl=sum(1 for r in recs if letter(r['text'],mode) is None) 16 secs=sum(r.get('secs',0) for r in recs); tok=sum(r.get('tokens') or 0 for r in recs) 17 rows.append((ok/max(n,1),model,mode,n,ok,nl,secs,tok)) 18for r in sorted(rows,reverse=True): print(f"{r[1]:28s} {r[2]:7s} n={r[3]:3d} right={r[4]:3d} ({100*r[0]:.1f}%) noletter={r[5]:3d} time={r[6]/60:.1f}min tokens/answer={r[7]/max(r[3],1):.0f}")
overlap.py counts how many models get each question right.
PYTHON
1import json,glob,re,sys,collections,math 2sys.path.insert(0,'.') 3from grade import letter 4M={} 5for p in sorted(glob.glob('runs/*__reason.jsonl')): 6 name=p.split('/')[1].split('__')[0] 7 recs=[json.loads(l) for l in open(p)] 8 if len(recs)<282: continue 9 M[name]={r['i']:(letter(r['text'],'reason')==r['key']) for r in recs} 10names=list(M); N=282 11print(len(names),'models with all 282 reasoning answers') 12cnt=[sum(M[n][i] for n in names) for i in range(N)] 13K=len(names) 14dist=collections.Counter(cnt) 15print('\nHow many of the',K,'models get each question right:') 16for k in range(K+1): print(f' {k:2d} right: {dist[k]:3d} questions', '#'*(dist[k]//2)) 17print('\nall right:',dist[K],' all wrong:',dist[0],' (',round(100*dist[K]/N),'% all right,',round(100*dist[0]/N,1),'% nobody)') 18# local only vs hosted 19loc=[n for n in names if not n.startswith('deepseek_')] 20cl=[sum(M[n][i] for n in loc) for i in range(N)] 21print('\nlocal models only (',len(loc),'): all right',sum(1 for c in cl if c==len(loc)),' all wrong',sum(1 for c in cl if c==0)) 22# pairwise agreement: P(both right)/ phi correlation 23def phi(a,b): 24 x=[M[a][i] for i in range(N)];y=[M[b][i] for i in range(N)] 25 n11=sum(1 for u,v in zip(x,y) if u and v);n10=sum(1 for u,v in zip(x,y) if u and not v);n01=sum(1 for u,v in zip(x,y) if not u and v);n00=N-n11-n10-n01 26 d=math.sqrt((n11+n10)*(n01+n00)*(n11+n01)*(n10+n00)) 27 return (n11*n00-n10*n01)/d if d else 0 28import itertools 29cors=[(phi(a,b),a,b) for a,b in itertools.combinations(names,2)] 30print('\nCorrelation of right/wrong between models (phi, 1 = identical pattern, 0 = unrelated):') 31print(' mean',round(sum(c[0] for c in cors)/len(cors),2),' min',round(min(cors)[0],2),min(cors)[1:],' max',round(max(cors)[0],2),max(cors)[1:]) 32# expected under independence: for each question P(right)=per-model accuracy; compare observed "nobody right" etc 33acc={n:sum(M[n].values())/N for n in names} 34exp_all_wrong=sum(math.prod(1-acc[n] for n in names) for i in range(N)) 35exp_all_right=sum(math.prod(acc[n] for n in names) for i in range(N)) 36print('\nIf models were independent, expect',round(exp_all_wrong,1),'questions nobody gets right and',round(exp_all_right,1),'all right; observed',dist[0],'and',dist[K]) 37json.dump({'cnt':cnt,'names':names},open('overlap.json','w'))
judge.py marks one model's answers.
PYTHON
1"""Judge the reasoning answers of one model. usage: judge.py JUDGE_MODEL ANSWER_FILE_STEM [limit] 2For every answer the judge sees the question, the options, the correct letter and the model's reply, and returns 3Accuracy, Completeness and Clarity (1 to 10) and a verdict: SOUND, SLIP, FLAWED or UNSUPPORTED.""" 4import json,sys,time,os,urllib.request 5judge,stem=sys.argv[1],sys.argv[2]; limit=int(sys.argv[3]) if len(sys.argv)>3 else 0 6orig=json.load(open('orig.json')) 7src=f"runs/{stem}__reason.jsonl" 8out=f"judged/{stem}__by__{judge.replace(':','_')}.jsonl" 9os.makedirs('judged',exist_ok=True) 10done={json.loads(l)['i'] for l in open(out)} if os.path.exists(out) else set() 11PROMPT="""You are marking a student's answer to a multiple-choice accounting question. You are given the correct answer. 12 13Question: 14{question} 15 16Options: 17{options} 18 19Correct answer: {key} 20 21Student's reply: 22\"\"\" 23{reply} 24\"\"\" 25 26Mark the reply. Give three scores from 1 to 10: 27- accuracy: is the reasoning and the figures right, and does it reach the correct answer? 28- completeness: does it work through what is needed to justify its answer? 29- clarity: is it easy to follow? 30And a verdict, one of: SOUND (right thinking), SLIP (a small error), FLAWED (wrong thinking or wrong answer), UNSUPPORTED (a claim with no reasoning behind it). 31Reply with JSON only: {{"accuracy": n, "completeness": n, "clarity": n, "verdict": "...", "reason": "one sentence"}}""" 32def ask(prompt): 33 body={"model":judge,"messages":[{"role":"user","content":prompt}],"stream":False,"format":"json","think":False,"options":{"temperature":0,"seed":1,"num_predict":300}} 34 req=urllib.request.Request("http://localhost:11434/api/chat",json.dumps(body).encode(),{"Content-Type":"application/json"}) 35 return json.load(urllib.request.urlopen(req,timeout=900))["message"]["content"] 36n=0 37with open(out,'a') as f: 38 for l in open(src): 39 r=json.loads(l) 40 if limit and n>=limit: break 41 n+=1 42 if r['i'] in done: continue 43 o=orig[r['i']] 44 opts="\n".join(f"{'ABCD'[k]}. {c}" for k,c in enumerate(o['choices'])) 45 t=time.time() 46 try: 47 raw=ask(PROMPT.format(question=o['question'],options=opts,key=r['key'],reply=r['text'][:6000])) 48 j=json.loads(raw) 49 rec={"i":r['i'],**{k:j.get(k) for k in("accuracy","completeness","clarity","verdict","reason")},"secs":round(time.time()-t,1)} 50 except Exception as e: 51 rec={"i":r['i'],"error":str(e)[:200]} 52 f.write(json.dumps(rec)+"\n"); f.flush() 53print('done',stem,judge)
plant.py is the planted-error test.
PYTHON
1"""Planted-error test: change one number in answers the checker marked right and the judge rated SOUND, then judge the damaged copy. 2The damage: the first figure after an equals sign has its last digit changed. The final ANSWER line is left alone.""" 3import json,random,re,urllib.request,glob,sys 4sys.path.insert(0,'.') 5from grade import letter 6from judge_prompt import PROMPT 7orig=json.load(open('orig.json')) 8def ask(prompt): 9 body={"model":"gemma4:12b","messages":[{"role":"user","content":prompt}],"stream":False,"format":"json","think":False,"options":{"temperature":0,"seed":1,"num_predict":300}} 10 req=urllib.request.Request("http://localhost:11434/api/chat",json.dumps(body).encode(),{"Content-Type":"application/json"}) 11 return json.loads(json.load(urllib.request.urlopen(req,timeout=900))["message"]["content"]) 12def damage(t): 13 m=re.search(r'=\s*\$?(\d[\d,]*\.?\d*)',t) 14 if not m: return None 15 num=m.group(1); i=m.start(1)+len(num)-1 16 while not t[i].isdigit(): i-=1 17 new=str((int(t[i])+3)%10) 18 return t[:i]+new+t[i+1:] 19cands=[] 20for mname in ['qwen3.8_27b-mlx','qwen2.5_72b','llama3.3_70b','gemma3_27b','qwen2.5_14b']: 21 runs={json.loads(l)['i']:json.loads(l) for l in open(f'runs/{mname}__reason.jsonl')} 22 for l in open(f'judged/{mname}__by__gemma4_12b.jsonl'): 23 j=json.loads(l); r=runs[j['i']] 24 if j.get('verdict')=='SOUND' and letter(r['text'],'reason')==r['key'] and damage(r['text']): cands.append((mname,j,r)) 25print(len(cands),'candidates') 26random.seed(3); pick=random.sample(cands,min(40,len(cands))) 27out=open('plant.jsonl','w') 28for m,j,r in pick: 29 o=orig[r['i']]; opts="\n".join(f"{'ABCD'[k]}. {c}" for k,c in enumerate(o['choices'])) 30 d=ask(PROMPT.format(question=o['question'],options=opts,key=r['key'],reply=damage(r['text'])[:6000])) 31 out.write(json.dumps({"model":m,"i":r['i'],"orig":{k:j[k] for k in('accuracy','verdict')},"damaged":d})+"\n") 32print('done')