Benchmarking TypeSafe's Jev at Battleship: the model tied with plain code

TypeSafe's Jev is not a chatbot. You hand it a state and some typed questions, and it hands back probability distributions. Ask it "which of these cells holds a ship" and you get a number for every cell, not a sentence.

That shape is interesting, so I wanted to know how good the judgement actually is. Battleship turned out to be a good way to find out. The rules fit on a napkin, the board is small enough to send in full, and there is a well-known non-model baseline sitting right there to measure against.

Sixty games per strategy, 6,000 model calls, 77 cents.

Two ways to ask

I built two model players.

The first one, which I called jevPure, is the honest measurement. It gets the raw board and every untried cell as an option, roughly 90 of them mid-game. Code supplies the rules and the list of legal moves and nothing else. Whatever it scores belongs to the model.

The second, jevHybrid, is a collaboration. A density solver counts every valid remaining-ship placement per cell, takes the best 16, and describes each one in plain words. Something like "directly continues 1 hit in a row below; has room for the longest ship still afloat; is in the open middle of the board". The model only picks between those 16.

Then three players that never call the model at all: random guessing as the floor, a hunt-and-target heuristic of the kind you would write in an interview, and the density solver on its own.

All five played the same 60 layouts, so every comparison is paired.

The scoreboard

Lower is better. Seventeen shots is a perfect game, 100 is the worst you can do.

Look at jevPure for a second. Given the whole board and 90 options, it needs 85.5 shots. Random guessing needs 95.3. So it is reading the board, and the gap is real, p is 1.3e-10.

But the hunt-and-target heuristic needs 51.9. The model lost to it by 33.6 shots and went down on 59 of the 60 boards. It cost 65 cents to finish barely ahead of random.

Why it fell over

Two reasons, both sitting in TypeSafe's own documentation, both of which I should have seen coming.

Every option in that Choice question is labelled identically. Cell A1 and cell H7 both come through as "a cell that has not been fired at yet", because there is nothing else honest to say about them. So the option list carries zero signal and all the work falls on parsing 2,791 characters of board state.

The second reason is more interesting. Jev rounds probabilities to two decimal places. Spread a distribution across 90 options and most of it rounds to zero. On a typical shot only six cells came back with any value at all. The heatmap I built to visualise the model's thinking was mostly empty, and that was not a bug in my rendering.

Give it 16 described options instead and the same model reaches 46.0 shots, comfortably past the heuristic. Same model, same board, different question.

The part I wanted to be true

Here is where I have to be careful with myself.

46.0 against the density solver's 48.3 looks like the model adding something on top of good code. That is the headline I wanted. It is also not supported by the data.

The confidence interval on that difference spans zero. The p value is 0.13. The model won 31 of 60 boards, which is a coin flip with extra steps. Sixty games can detect a difference of about 4.7 shots and the gap I measured is half of that.

Mean shots to sink the fleet, with 95% confidence intervals. The overlap between jevHybrid and Density is the result.

So the honest claim is that jevHybrid matches the best code-only player rather than beating it. Settling it properly needs around 260 games per strategy, which is two and a half hours and another 80 cents. I will probably run it. But I am not going to describe a tie as a win in the meantime.

What I can say is narrower and still useful. The model does real work, and that work shows up only once code has already turned the problem into a shortlist of described options. Asked to find a ship on a raw board it is far worse than a rule you could write from memory. Which means the configuration where the model looks good is the same configuration where the code did the hard part.

The bias I nearly shipped

The result that actually changed how I will build benchmarks had nothing to do with the model.

I had been generating fleet layouts uniformly at random, which felt obviously neutral. Then I tried other families. Ships pushed against the edges. Ships kept away from the edges. Ships placed greedily on the cells a density map rates lowest.

The ranking between my two code baselines inverted.

The mechanism is not subtle once you see it. The density solver counts placements under a uniform prior, so on an empty board it rates the middle at 28.5 and the edge at 18.2 and searches the middle first. An edge layout puts all 17 ship cells exactly where it looks last.

Uniform-random layouts are not a neutral test set. They are the one distribution the density solver assumes. And the density solver is the bar I was holding the model to, so testing only on random layouts would have meant judging the model against an opponent playing on home ground, without either of us noticing.

If your baseline has a prior baked into it, and most good baselines do, check whether your test set happens to match it. I now run everything on a mixed set and every result records which family it used.

What I would tell you to copy

Build the ladder before you build the model player. Random at the floor, a naive heuristic, then the strongest plain-code solution you can be bothered to write. A single model score means nothing without them, and the strongest baseline is the one that keeps you honest.

Run the significance test even when you like the answer. Especially then. I had a 2.3-shot lead and a story to go with it, and the arithmetic said coin flip.

Keep all the arithmetic in code. Jev is documented as unreliable at counting, and Battleship is a counting game underneath, so every placement count and every probability calculation lives on my side of the API. The model is only ever asked to judge between options that code has already proved are legal. That constraint is what makes jevHybrid work at all.

And write down what the numbers do not establish, in the same place as the numbers. My results file says jevHybrid matches density and names the sample size that would settle it. In six months I will have forgotten, and that line is the only thing standing between me and quoting my own tie as a win.

The full write-up, with the paired statistics, the three board representations I compared, and the cost breakdown, is here.

The code is open source at github.com/ickas/battleship-vs-jev: the engine, the five strategies, the benchmark runners and the 228 tests. npm run bench reproduces the table above.