Our benchmarks

A game

Every move is a fresh request. The model gets the variant, its colour, the starting position, the moves so far and the current position, and answers with one move inside a tag. It can think as long as its settings allow. It doesn't get a list of legal moves, because keeping track of the board is part of the test.

An illegal or unreadable answer gets one line of feedback and another try, up to 3 tries a move. After that the model forfeits. Every rejected try is kept and counted under Illegal / invalid. The denominator is answered requests, including retries, from finished games. Transport failures are excluded.

A game ends normally (mate, stalemate, repetition, the 50-move rule), or by adjudication: once Stockfish rates the position at 10 pawns or more for one side across 6 measured positions in a row, after a long dead-level stretch, or at 300 plies. Adjudicated games are flagged.

Ratings

Models only ever play the opponents in the ladder, never each other. Every game is one result against an opponent with a fixed benchmark rating, and each model's rating is fitted to its own results alone. Adding a model never moves anyone else's number. A Chess960 game uses the same start position for every model, in the same order. Colours alternate.

We pin Stockfish's limited-strength levels at their nominal UCI_Elo values and fit the weaker opponents against them. This is an internal engine-anchored scale. We haven't independently validated those nominal values at our 0.5-second move time or in Chess960. It isn't interchangeable with ChessBench, FIDE or Chess.com ratings.

Grading

Stockfish analyses the position before and after every model move at 1,000,000 nodes. From that we take the engine's best move, its top three, and how much winning chance the move gave up. Inaccuracies, mistakes and blunders use Lichess's thresholds (5, 10 and 15 points of winning chance). Accuracy is Lichess's per-move formula. Because that formula forgives every move in a position already won or lost, we report it only for moves made while the game was still open.

Cost and time

Claude models run through Claude Code and OpenAI models through the Codex CLI, on a subscription, with our own instructions and every tool switched off. Local models run on our own DGX Sparks. Nobody is billed per request, so the cost we show is API-equivalent: what the same tokens, rejected tries included, would cost at the lab's list price. Local models have no list price, so their cost shows as n/a, not zero.

Token counts are what each request actually used. The Codex CLI adds about 9,800 tokens of its own context to every request and Claude Code about 640, so input tokens are higher than the chess prompt alone. Time is the wall-clock wait for each answer, including any queue on our side, so compare it within a route rather than across them.

What it doesn't show

Chess is one skill. A strong rating says a model can track a board and plan within these rules. It doesn't say how good the model is at your work. Models with no reasoning setting run at their provider's default. A request that fails on our side (a timeout, an outage, a usage limit) aborts the game, and an aborted game never counts against the model.

The grids

Every grid comes from Cozy Sudoku's generator, and every one is checked to have exactly one solution. The first six tiers are the game's own difficulty bands. A grid's band is set by the hardest technique it needs, not by how many digits are given. The seventh tier, Beyond, holds grids that none of those techniques finishes. We generated 400 of them and kept the ones that need the most trial and error.

Every model gets the same 6 grids per tier from set v1. The grids are generated from fixed seeds, not taken from a puzzle book, so a model can't have memorised them. The set is published with the results, so when it's been around long enough to be trained on, we'll move to a new one and label it.

An answer

The model gets the grid once, as nine rows, and answers with the whole solution and a confidence from 0 to 100. It can think as long as its settings allow. There are no hints and no second try at a wrong answer. The one retry is for a reply with no readable grid in it, which is a formatting slip, not an answer.

We check every cell against the solution. An answer is either solved, wrong in some cells, left with cells empty, or one that changed a given digit. Cells right counts every blank cell filled correctly, so a near miss still shows up. Broke the rules counts complete grids that repeat a digit somewhere: the model said it was done without checking. On a model's own page, cell accuracy is split by the technique each cell needed.

Score

The score is the solve rate in each tier, averaged so every tier counts the same, from 0 to 100. Its 95% interval adds up each tier's uncertainty. With 6 grids a tier, two models a few points apart are not reliably different.

Confidence

We ask for a confidence on every answer and score it with the Brier score: the squared gap between what it said and what happened, averaged. 0 is perfect. Always saying 50% scores 0.25. Sure but wrong counts unsolved answers it called 90% certain or more.

Cost and time

Same as the chess benchmark: Claude models run through Claude Code and OpenAI models through the Codex CLI on a subscription, with every tool switched off, and local models on our own DGX Sparks. Cost is API-equivalent, what the tokens would cost at the lab's list price. Local models show n/a. Time is the wall-clock wait for the answer. A request that fails on our side is never counted against the model.