Jev vs Clef at chess: 300 games, one fair request, a lot of draws

In September I put TypeSafe's Jev on a Battleship board, and last week I ran Cloudflare's Clef through the same benchmark. Both times the result had the same shape. Each model tied with plain code once code had built it a shortlist, and on a raw board every one of them was far behind a few lines of heuristic.

Battleship has one problem as a benchmark for these models: nobody plays back. The models never faced each other, only a fleet that sits still. So the next question was obvious. What happens model against model, in a game where the opponent punishes every mistake?

I built benchessmark, had Jev, Clef and Clef-flash play 300 games of chess, and let Stockfish grade every move.

Game 31: Jev (white) against Clef, rendered straight from the game file. The arrows are each model's top three candidate moves.

Two models that don't write

If you missed the earlier posts: Jev and Clef aren't chat models. You send them a state and a set of typed questions, and instead of writing an answer they return a probability for every option you allowed. One pass, no text. Clef is the 27B model, and Clef-flash is the 9B one, built for speed.

That shape is a gift for chess. Each turn, I ask one question ("Pick the best move for white in this position") and pass every legal move as an option. The most legal moves any chess position can have is 218, and both APIs accept up to 255 options per question, so a whole position always fits. It also means a model can never play an illegal move. Anyone who has watched a chat model try to play chess knows how much time that saves.

Making it fair when there's nothing to tune

My first instinct was to line up the settings. Same effort, same temperature, same everything, so neither model gets a head start.

There's nothing to line up. Neither API has an effort setting, a temperature or a seed. The model reads the state once and answers.

So fairness has to come from the harness. Both models go through the same endpoint (OpenRouter's Decisions API), so the network path is identical and the cost is billed instead of estimated. Both get the same state: the FEN, a text board with every square spelled out, and the move list so far. The text board is there because TypeSafe's own docs list counting as a weak spot for Jev, and a FEN rank like n2Pp1N1 has to be counted to be read.

Before game one, the runner builds each player's first request and compares them. If anything but the model id differs, the match doesn't start. It prints the comparison side by side, which turned out to be the most reassuring screen in the whole project:

                      Jev               Clef
transport             openrouter        openrouter       ✓
question              #ee21d4           #ee21d4          ✓
pick                  top, seeded tie   top, seeded tie  ✓
effort, temp          none exist        none exist       ✓
first request         same except model #273b9f          ✓

Not settable
probability decimals  2                 4                !
build reported        resolved build    alias only       !
state read limit      32k tokens        ≈2k tokens       ✓

That last block is the part I couldn't fix, so I print it instead. Jev rounds its probabilities to two decimals and Clef returns four, which I'd already measured in the Battleship project. Jev tells you exactly which build answered (jev-1.13-20260917) and Clef only reports its alias. And Clef's serving stack reads roughly the first 2,000 tokens of the state. A long chess game in this format is about 360 tokens, and every turn checks it stays under 1,500, so that one is handled.

The rule for turning probabilities into a move is the same for both: play the most probable legal move, and break ties with a seeded draw. I don't use the API's own choice field because its tie rule isn't documented, and with Jev's two decimals, ties at the top happen in about 5% of moves.

Every match plays the same 50 opening lines (named from the lichess openings list), once with each model as white. That's 100 games per pair. Stockfish 19 then grades every move at depth 18, and accuracy uses the same formula lichess uses.

The results

Both Clefs beat Jev, and neither interval comes close to zero. Against each other, the two Clefs are a coin flip: Clef-flash scored a little more, but 100 games can't separate them.

Move by move, the differences are smaller than the scores suggest. Against Jev, Clef played at 63.5% accuracy to Jev's 58.5%, with an average centipawn loss of 101 against 106. Both blundered about 9 times per 100 moves. Both found Stockfish's best move less than a third of the time. If you're a chess player, that's a club player who moves instantly and never calculates, which is more or less exactly what a single-pass model is.

Where they really differ: knowing when they're right

Every answer comes with a confidence score, and this is where the models split.

I grouped every move by the model's confidence and checked how often it played Stockfish's move:

When the Clefs are sure, they're right almost every time, and they're sure about a hundred times per match. Jev reached that level 4 times in 100 games. At low confidence, all three look the same.

For anyone building on these models, that table matters more than the Elo gap. TypeSafe's docs suggest acting above a confidence threshold and escalating below it. With Clef, that strategy has a lot of room to work. With Jev, almost everything lands below the line.

Speed and money

Jev is the fastest and cheapest, by a lot. Clef costs about five times more per game. Clef-flash plays the same chess as Clef at about a third of the price, which makes it the obvious pick if you only look at these numbers.

The three 100-game matches together cost about $2.50. The whole benchmark, including the test runs, came in under $3.

So many draws

I didn't see this one coming. Between 76 and 83 games out of every 100 ended in a draw, and almost all of them by threefold repetition.

It makes sense once you think about how these models play. They don't plan. They see the position, they score the moves, and the harness plays the top one. In a quiet position, the model gives the same answer every time it sees that position, so the pieces shuffle back and forth until the position repeats three times. The move list is in every request, so the models could in principle notice. None of them did.

I could make this go away by sampling from the probabilities instead of always taking the top move, but that changes the experiment. Jev's two-decimal rounding sets many options to exactly zero, so sampling would treat the two models differently. I'd rather report the draws honestly than hide them with a setting that quietly favours one model.

Watching the games

I wanted to see the games, not just the numbers. benchessmark has a replay viewer that shows the board, each model's top three candidate moves as arrows with their probabilities, the Stockfish evaluation and every move's grade on a timeline. It can follow a match live while it plays.

For recording, I first thought about capturing my screen. Instead, the viewer has a record mode, and a script steps through it frame by frame in a headless browser and pipes every frame to ffmpeg. The same game always produces the same video, at any speed, and a 50-second clip renders in a bit over a minute.

The game at the top of this post is a good example of how these models play. Jev, as white, gets a clear advantage out of a Sicilian Rossolimo, then gives it back with 9. d4 and drops its queen a move later. Clef doesn't hesitate: it takes everything on offer and mates on move 18 with Qxe1#. Watch the arrows, and you can see how rarely either model is sure of anything until the position gets simple.

What this doesn't tell us

The same caveats I put in every benchmark write-up apply here.

This is one setup: one question, one state format, the top move always played, no lookahead. A different prompt or a different state could rank the models differently. It doesn't say Clef is better than Jev in general. It says Clef plays better chess in this setup.

It doesn't rank Clef against Clef-flash. Their interval includes zero.

It only covers the builds that answered. Jev was jev-1.13-20260917 in every call. The Clefs only report their alias, so a silent update during these runs wouldn't show up anywhere.

And more runs wouldn't help much. With the top move always played, rerunning the same openings would most likely replay the same games. More evidence needs more openings, not more reruns.

Try it

The code, all 300 games as PGN, and the full results write-up are on GitHub. Adding another System One model is one entry in a config file, and the harness ships with two code players (a random mover and Stockfish held to 1320 Elo) so you can place any new model on the same scale.

In Battleship, every model needed code to do half the work before it could keep up. In chess, nobody did the work for them, and the models played like what they are: fast, cheap, instinctive, and with no idea what they're going to do two moves from now.