BENCH / LOGIC & RELIABILITY

Sudoku benchmark

Name: Model Cadence Bench: Sudoku
Creator: Model Cadence

Shared grids across seven difficulty tiers, from Cozy Sudoku's engine. One answer, no hints. Compare solve rates, confidence, cost and response time.

14 model settings391 answers249 solvedUpdated 2026-10-25

Methodology Download CSV ↓

Sudoku leaderboard

#	Model	Score	Solved per tier (of 6)CozGenSteTriDevExpBey	Cells right	Broke the rules	Sure but wrong	Calibration	API cost / grid	Output tokens / grid	Sec / grid
1	Claude Opus 5.5reasoning medium	100	6666666	100.0%	0	0 of 0	0.000	$0.244	11,827	101
2	Claude Fable 5.1reasoning medium	93	6664566	97.6%	2	2 of 3	0.047	$0.894	16,999	173
3	GPT-6 Solreasoning medium	90	6666464	93.7%	2	2 of 4	0.048	$0.149	12,925	142
4	Claude Sonnet 5.5reasoning medium	88	5556466	99.5%	1	5 of 5	0.111	$0.138	13,462	91
5	GPT-6.1 Solreasoning medium	83	6645545	94.1%	7	5 of 7	0.119	$0.112	8,723	195
6	GPT-6 Astrareasoning medium	76	5454464	94.0%	9	10 of 10	0.238	$0.453	6,987	226
7	Claude Haiku 5.5reasoning medium	31	3113221	93.1%	8	29 of 29	0.643	$0.015	30,157	111
8	GPT-6 Lunareasoning medium	0	0000000	8.6%	3	3 of 42	0.075	$0.0024	1,779	24
9	Qwen3.8 Flash-Nextlocal, thinking low · open weights	5022 of 42	2311220	48.9%	0	0 of 11	0.000	n/a	64,680	3,426
10	gpt-oss 20Blocal, thinking low · open weights	405 of 42	01100··	74.7%	0	2 of 3	0.500	n/a	24,740	2,702
11	Qwen3.8 27Blocal, no thinking · open weights	07 of 42	0000000	29.5%	2	4 of 7	0.593	n/a	12,309	1,325
12	Ministral 3 8Blocal, no thinking · open weights	07 of 42	0000000	12.4%	0	7 of 7	1.000	n/a	108	20
13	Qwen3.5 9Blocal, no thinking · open weights	07 of 42	0000000	9.3%	0	0 of 7	0.000	n/a	2,402	242
14	Gemma 3 4Blocal, no thinking · open weights	07 of 42	0000000	6.7%	0	7 of 7	0.925	n/a	177	18

Scroll for all metrics. Score is the solve rate averaged over the seven tiers, each tier weighted the same, from 0 to 100. A model still working through the set shows how many of the 42 grids it has answered, and ranks after the models that have finished. Click a column to sort.

Cost frontierAPI-equivalent cost per grid. Local models have no list price and are omitted. Bars show 95% intervals.

Closed weightsOpen weightsPareto frontier

Time frontierObserved response time, including reasoning and queue wait. Routes and hardware differ.

Closed weightsOpen weightsPareto frontier

The Pareto frontier shows where no other model scores at least as well for less cost or time, with one strict improvement. Partial sets are included and may cover different puzzles. Use completed sets for direct comparisons.

Individual answers and confidence

Confidence and solve rateBelow the line: stated confidence exceeds the observed solve rate.

Closed weightsOpen weightsConfidence matches results

Output tokens by difficulty

Average output tokens per grid, reasoning included. Token counts include reasoning when reported by the provider.

Model	Cozy	Gentle	Steady	Tricky	Devious	Expert	Beyond
Claude Opus 5.5	4,521	8,296	12,067	16,782	8,398	11,761	20,962
Claude Fable 5.1	5,128	10,788	16,626	11,143	9,188	23,410	42,711
GPT-6 Sol	4,229	9,606	11,675	15,702	7,839	12,594	28,828
Claude Sonnet 5.5	5,884	10,288	14,392	17,670	8,807	14,058	23,134
GPT-6.1 Sol	3,938	6,007	8,432	8,794	6,595	9,598	17,701
GPT-6 Astra	2,768	4,133	10,358	5,546	8,886	5,917	11,303
Claude Haiku 5.5	15,853	22,690	34,434	33,600	20,784	38,906	44,833
GPT-6 Luna	1,826	1,485	1,971	1,748	1,836	2,203	1,381
Qwen3.8 Flash-Next	46,297	36,521	91,260	72,341	54,011	64,097	94,361
gpt-oss 20B	8,212	18,046	16,661	64,841	15,941	n/a	n/a
Qwen3.8 27B	18,402	10,077	11,476	11,251	10,583	9,257	15,116
Ministral 3 8B	108	108	108	108	108	108	108
Qwen3.5 9B	3,000	3,000	3,000	3,000	1,606	1,605	1,605
Gemma 3 4B	142	295	133	259	132	149	127

Every grid

One square per answer, filled when it solved the grid. Columns run from the easiest tier to the hardest. Click a square to see the answer.

Model	coz	2	3	4	5	6	gen	2	3	4	5	6	ste	2	3	4	5	6	tri	2	3	4	5	6	dev	2	3	4	5	6	exp	2	3	4	5	6	bey	2	3	4	5	6
Claude Opus 5.5
Claude Fable 5.1
GPT-6 Sol
Claude Sonnet 5.5
GPT-6.1 Sol
GPT-6 Astra
Claude Haiku 5.5
GPT-6 Luna
Qwen3.8 Flash-Next
gpt-oss 20B
Qwen3.8 27B
Ministral 3 8B
Qwen3.5 9B
Gemma 3 4B

MethodologyProtocol, scoring & limitations

The grids

Every grid comes from Cozy Sudoku's generator, and every one is checked to have exactly one solution. The first six tiers are the game's own difficulty bands. A grid's band is set by the hardest technique it needs, not by how many digits are given. The seventh tier, Beyond, holds grids that none of those techniques finishes. We generated 400 of them and kept the ones that need the most trial and error.

Tier	Hardest technique needed	Givens
Cozy	Singles only	31 to 42
Gentle	Scanning	25 to 30
Steady	Locked candidates	27 to 32
Tricky	Hidden pairs and triples	24 to 27
Devious	Wings and fish	26 to 41
Expert	Deep wings and fish	23 to 28
Beyond	No human technique finishes it	21 to 25

Every model gets the same 6 grids per tier from set v1. The grids are generated from fixed seeds, not taken from a puzzle book, so a model can't have memorised them. The set is published with the results, so when it's been around long enough to be trained on, we'll move to a new one and label it.

An answer

The model gets the grid once, as nine rows, and answers with the whole solution and a confidence from 0 to 100. It can think as long as its settings allow. There are no hints and no second try at a wrong answer. The one retry is for a reply with no readable grid in it, which is a formatting slip, not an answer.

We check every cell against the solution. An answer is either solved, wrong in some cells, left with cells empty, or one that changed a given digit. Cells right counts every blank cell filled correctly, so a near miss still shows up. Broke the rules counts complete grids that repeat a digit somewhere: the model said it was done without checking. On a model's own page, cell accuracy is split by the technique each cell needed.

Score

The score is the solve rate in each tier, averaged so every tier counts the same, from 0 to 100. Its 95% interval adds up each tier's uncertainty. With 6 grids a tier, two models a few points apart are not reliably different.

Confidence

We ask for a confidence on every answer and score it with the Brier score: the squared gap between what it said and what happened, averaged. 0 is perfect. Always saying 50% scores 0.25. Sure but wrong counts unsolved answers it called 90% certain or more.

Cost and time

Same as the chess benchmark: Claude models run through Claude Code and OpenAI models through the Codex CLI on a subscription, with every tool switched off, and local models on our own DGX Sparks. Cost is API-equivalent, what the tokens would cost at the lab's list price. Local models show n/a. Time is the wall-clock wait for the answer. A request that fails on our side is never counted against the model.

Data: JSON · CSVUse with credit to Model Cadence.