Chess benchmark

A game

Every move is a fresh request. The model gets the variant, its colour, the starting position, the moves so far and the current position, and answers with one move inside a tag. It can think as long as its settings allow. It doesn't get a list of legal moves, because keeping track of the board is part of the test.

An illegal or unreadable answer gets one line of feedback and another try, up to 3 tries a move. After that the model forfeits. Every rejected try is kept and counted under Illegal / invalid. The denominator is answered requests, including retries, from finished games. Transport failures are excluded.

A game ends normally (mate, stalemate, repetition, the 50-move rule), or by adjudication: once Stockfish rates the position at 10 pawns or more for one side across 6 measured positions in a row, after a long dead-level stretch, or at 300 plies. Adjudicated games are flagged.

Ratings

Models only ever play the opponents in the ladder, never each other. Every game is one result against an opponent with a fixed benchmark rating, and each model's rating is fitted to its own results alone. Adding a model never moves anyone else's number. A Chess960 game uses the same start position for every model, in the same order. Colours alternate.

We pin Stockfish's limited-strength levels at their nominal UCI_Elo values and fit the weaker opponents against them. This is an internal engine-anchored scale. We haven't independently validated those nominal values at our 0.5-second move time or in Chess960. It isn't interchangeable with ChessBench, FIDE or Chess.com ratings.

Grading

Stockfish analyses the position before and after every model move at 1,000,000 nodes. From that we take the engine's best move, its top three, and how much winning chance the move gave up. Inaccuracies, mistakes and blunders use Lichess's thresholds (5, 10 and 15 points of winning chance). Accuracy is Lichess's per-move formula. Because that formula forgives every move in a position already won or lost, we report it only for moves made while the game was still open.

Cost and time

Claude models run through Claude Code and OpenAI models through the Codex CLI, on a subscription, with our own instructions and every tool switched off. Local models run on our own DGX Sparks. Nobody is billed per request, so the cost we show is API-equivalent: what the same tokens, rejected tries included, would cost at the lab's list price. Local models have no list price, so their cost shows as n/a, not zero.

Token counts are what each request actually used. The Codex CLI adds about 9,800 tokens of its own context to every request and Claude Code about 640, so input tokens are higher than the chess prompt alone. Time is the wall-clock wait for each answer, including any queue on our side, so compare it within a route rather than across them.

What it doesn't show

Chess is one skill. A strong rating says a model can track a board and plan within these rules. It doesn't say how good the model is at your work. Models with no reasoning setting run at their provider's default. A request that fails on our side (a timeout, an outage, a usage limit) aborts the game, and an aborted game never counts against the model.