An audit of boar — one month, 387 experiments, a 1.7B model in a 3 GB APK with no internet permission. Everything below is recomputed from the author's published artefacts or measured in a run of my own; tags say which.
The finding, in one sentence: the headline "0 of 71 wins" is an artefact of taking one sample per item — the single item that was not a loss in the published Codex-family run is a stable win for the phone app, 3 out of 3, under the same judge model, when the judge is allowed to sample.
That does not rescue the app: the aggregate gap to the 0.50 bar is 5–10× larger than any judge noise I could measure, and it survives everything in this audit. It does mean the study is presenting a statistic with no resolution as its most quoted number, while the honest version — "at most about 4% of items win" — is both weaker and more defensible.
| Judge family | n | U-total | 95% CI | win | tie | loss |
|---|---|---|---|---|---|---|
| G1 Claude Opus vs Claude refs | 71 | 0.1668 | 0.129–0.209 | 0 | 0 | 71 |
| G2 gpt-5.6-sol vs Codex refs | 71 | 0.2440 | — | 0 | 1 | 70 |
From arms/device-final/aggregates/m1.json. Note the second row: it is not 0 losses-plus-zero, it is 70 losses and one order-split (win_tie_rate 0.00704). Recomputing the 150 per-order verdicts in dev-s42.codex.jsonl gives the same: 74 item-orders lost, 1 split, and the split is cw-038 — reference in order A, phone in order B.
I reconstructed the published judge pipeline (judge-core.mjs: system prompt, pairwise template, blind ordering) and re-judged the four closest losses — the boundary where a count of zero is decided — using the real published pairs: app answers from df-m1-s42.jsonl, references from bench/v0.4/refs/v2/claude.jsonl (sha-verified content), 2 presentation orders × 3 repeats × 2 judge models, temperature 1.0. 48 calls, all parsed.
| Item | Published verdict (G2) | gpt-5.6-sol, 3 repeats | Opus, 3 repeats |
|---|---|---|---|
| cw-038 | order-split (counted as tie) | win · win · win | loss ×3 |
| cw-020 | loss | win · tie · loss | loss ×3 |
| cw-011 | loss | loss ×3 | loss ×3 |
| cw-014 | loss | loss ×3 | loss ×3 |
A win requires the judge to prefer the app in both presentation orders. The reference it has to beat scores an absolute 3.874 / 5 on the same rubric (abs_ref, G1 aggregates) and is written by the same model family that judges it. Under that ceiling the win condition is close to structurally unattainable: on the earlier v0.2 ruler, every configuration — including the desktop deep+thinking tier that is an order of magnitude larger — also scored exactly 0 wins. A metric that returns zero for a 35B model and for a 1.7B model is not measuring the 1.7B.
What I would publish instead. Lead with U and its interval: 0.167 [0.129–0.209], gap to the bar 0.33 — eight times the CI width, five to ten times the judge noise I measured. Then state the win-rate honestly as "≤ 4% of 71 items (95% bound); 1 item wins stably under re-judging at temperature 1.0". Same conclusion, no free ammunition for anyone who wants to dismiss the record.
387 registry rows resolve to about 250 independent hypotheses; 194 of them are judged screens. The frozen promotion rule (wins ≥ control+3 on 20-item slices, or Δ ≥ +0.03 at SE 0.045) implies a per-test α of about 0.25 — I checked both arms of it (P(wins≥12 | n=20, null) = 0.2517). The escalation ladder catches most of them: SEFIX-ALL gained +0.023 on the screen and was withdrawn when cross-validation gave −0.002. But of the 52 positive rows carrying a CI, Benjamini–Hochberg at q=0.05 leaves 19, and 4 of the 9 per-pack deltas (travel, health, outdoors, firstaid) do not survive a Bonferroni at k=9. The pooled Δ does not depend on them; the prose about individual packs does.
PACKPROOF's pre-registered criterion reads sentences must use only packs that PROVE; the count was 0 of 9. The close-out report then introduces a weaker "conditional prove" rule and names 4 packs as satisfying it — a rule that appears in no criterion file — while one of those four (qa-health-se) has a pooled Δ of +0.05 with a CI of −0.15 to +0.25. That is the one place in the record where a number outruns its own rule.
The registry splits it as 187 closed + 28 "not detected", and the registry's own definition says not-detected is not evidence of no effect; a parity arm is excluded from the count entirely. The honest phrasing is "about 250 hypotheses tested; 187 closed by their own frozen rule" — which is a stronger statement about method and a weaker one about the null.
The 4B model was killed on memory, not on quality — and the memory problem looks like a repack artefact rather than a phone limit:
Run: Qwen3-4B at Q3_K_S or IQ3_XXS, q8_0 KV, mmap-backed weights with no anonymous repack, ctx 2048, two seeds. GO iff PSS ≤ 4.7 GB, zero LMK kills, bench ratio ≥ 0.60, p50 ≤ 40 s. Falsifier: ratio < 0.56 or PSS still over budget with mmap — then the drop was correct and this audit is wrong about it. Expected: bench +0.08 to +0.12, ruler U plausibly 0.165 → 0.25–0.35.
github.com/r4topunk/boar-bench; registry from boar-research; traces from the r4topunk/boar-traces dataset. Headline aggregates recomputed with the author's own m1_agg.py and matched to m1.json. My judge run used the gateway API with the reconstructed prompt, not the Codex CLI harness, and the run's frozen references file is not published (I used the repo's pinned copy, content-consistent) — so the 3/3 result on cw-038 should be read as the published judge model, sampling on, reproduced outside the original harness, not as a byte-identical rerun. Judge model id used: anthropic/claude-opus-5-5 (the one recorded in the run; the 4.6 id returns 404 on that gateway) and openai/gpt-5.6-sol. Raw verdicts: 48 rows, judgments.jsonl. Tags: verified read in code/artefacts · measured produced by this audit · recommendation untested proposal.