Benchmark report
TypeSafe's Jev is not a generative model. You hand it a state and typed questions, and it hands back probability distributions. So: can it play Battleship?
Battleship is a good test for a judgement model. The rules are trivial, the state is small, and there is a strong non-model baseline to measure against — so any credit the model takes has to be earned against code that already plays well.
Jev answers three kinds of question: a Choice over named options, a Score against ordered levels, and a boolean probability. Every answer carries a distribution, not just a pick. For Battleship that is the whole game: ask which cell holds a ship, and the per-option probabilities are a heatmap of where the model thinks the fleet is.
The design constraint comes from TypeSafe's own documentation. Jev is not a calculator, it loses accuracy as irrelevant state grows, and it does better on semantic descriptions than on positions it has to derive. So every rule check, every placement count and every probability calculation lives in code. The model is only ever asked to judge between options code has already established are legal.
One axis runs through all of them: how much of the thinking is code, and how much is the model.
| Strategy | Model calls | What it does |
|---|---|---|
| Random | 0 | Uniform among untried cells. The floor. |
| Hunt / Target | 0 | Parity search, then works outwards from a hit. |
| Density | 0 | Counts every valid remaining-ship placement per cell. The strongest code player. |
| Jev pure | 1 / shot | One Choice over every untried cell, from raw board state. |
| Jev hybrid | 1 / shot | Code ranks the top 16 and describes each; the model compares those. |
Both strategies face the position below. A ship has been hit at H8 and not sunk; another was sunk at F5–F7. Jev pure is handed all 88 untried cells as options, every one labelled identically — "A cell that has not been fired at yet." All the signal sits in 2,791 characters of board state it must parse unaided.
Jev hybrid is handed 16, each described in words: "directly continues 1 hit in a row below; has room for the longest ship still afloat; is in the open middle of the board."
60 games per strategy, all five playing the same 60 layouts, so every comparison is paired. Lower is better: 17 shots is a perfect game, 100 is the worst possible.
| Strategy | Mean | Median | SD | Best | Worst | Acc. | Tokens / game | Cost |
|---|---|---|---|---|---|---|---|---|
| Jev hybrid (top 16) | 46.0 | 47.0 | 9.6 | 26 | 71 | 37% | 54,365 | $0.12 |
| Probability density | 48.3 | 49.0 | 11.6 | 25 | 69 | 35% | — | — |
| Hunt / Target | 51.9 | 52.0 | 7.6 | 33 | 64 | 33% | — | — |
| Jev pure (per-cell list) | 85.5 | 88.5 | 12.2 | 48 | 100 | 20% | 297,315 | $0.65 |
| Random | 95.3 | 97.0 | 5.3 | 69 | 100 | 18% | — | — |
| Paired comparison | Difference | 95% CI | p | Verdict |
|---|---|---|---|---|
| Jev hybrid − Density | −2.22 | [−5.11, 0.68] | 0.13 | not distinguishable |
| Jev hybrid − Hunt/Target | −5.90 | [−8.96, −2.84] | 0.00015 | significant |
| Density − Hunt/Target | −3.68 | [−7.47, 0.10] | 0.056 | not distinguishable |
| Jev pure − Random | −9.78 | [−12.76, −6.80] | 1.3e−10 | significant |
| Jev pure − Hunt/Target | +33.57 | [30.08, 37.05] | <1e−15 | significant |
Choosing from ~90 cells is close to guessing.
Given the raw board and every untried cell as an option, Jev needs 85.5 shots against random's 95.3. The 9.8-shot gap is real — it is reading the board — but it loses to Hunt/Target, a fifty-line heuristic, by 33.6 shots, and lost on 59 of 60 layouts. It cost $0.65 to finish barely ahead of random.
Two documented traits explain it. Every option is labelled identically, so nothing in the option list helps. And probabilities come back rounded to two decimals, so spread across ~90 options most of the distribution collapses to zero — on a typical shot only a handful of cells carry any value at all.
With a shortlist, it matches the code baseline — but does not beat it.
With code ranking the top 16 and describing each in words, Jev reaches 46.0 shots, comfortably past Hunt/Target (p = 0.00015). Against Density it is nominally ahead by 2.2 shots — but the confidence interval spans zero, p is 0.13, and it won 31 of 60 layouts. That is a coin flip.
So the honest reading is that Jev hybrid matches the best code-only player rather than beating it. This run could detect a 4.7-shot difference; the observed gap is half that. Settling it needs roughly 260 games per strategy — about two and a half hours and $0.80.
Put together: the model does real work, but its contribution shows up only once code has reduced the problem to a short list of described options. Asked to find a ship on a raw board, it is far worse than a simple hand-written rule. That is exactly consistent with the documented jaggedness — and it means the configuration where Jev looks good is also the one where code has done the hard part.
The interface exposes exactly the dimensions that change what is being measured. Each one is in the URL, so a link reproduces a run.
The most surprising result was not about the model at all. Density counts placements under a uniform prior, so uniform-random layouts are the one case it is built for. Measured over 200 layouts per family, the ranking between the two code baselines inverts:
| Layout family | Density | Hunt / Target | Gap |
|---|---|---|---|
| Random | 43.4 | 51.8 | density +8.3 |
| Edge | 52.3 | 50.1 | density −2.2 |
| Adversarial | 53.0 | 49.8 | density −3.1 |
| Mixed | 48.6 | 48.7 | level |
This is a measurement bias, not a bug. Benchmarking only on uniform-random layouts
tests density on precisely the distribution it assumes — and density is the bar the
model is held to. The benchmark above therefore runs on mixed, and
every result records its family.
The game engine is pure deterministic code with no model involvement, covered by 226 unit tests. Every model call goes through a single client that logs the state, the questions, the response, latency, tokens, confidence and the model version.
jev-1.13.0
only. Calling TypeSafe directly reports the build that answered; through Vercel
AI Gateway only the requested alias comes back, so a silent model update would
be invisible.mixed. Results are not
comparable across families.To reproduce:
npm run bench -- --transport direct --layouts mixed --games 60 \ --strategies random,huntTarget,density,jevPure,jevHybrid --seed 1000
Every call's generation id is written to the JSON output, so any row in the results can be traced back to a specific request in the provider's logs.