The zero is not a zero

An audit of boar — one month, 387 experiments, a 1.7B model in a 3 GB APK with no internet permission. Everything below is recomputed from the author's published artefacts or measured in a run of my own; tags say which.

The finding, in one sentence: the headline "0 of 71 wins" is an artefact of taking one sample per item — the single item that was not a loss in the published Codex-family run is a stable win for the phone app, 3 out of 3, under the same judge model, when the judge is allowed to sample.

That does not rescue the app: the aggregate gap to the 0.50 bar is 5–10× larger than any judge noise I could measure, and it survives everything in this audit. It does mean the study is presenting a statistic with no resolution as its most quoted number, while the honest version — "at most about 4% of items win" — is both weaker and more defensible.

1. What the published record actually says verified in artefacts

Judge familynU-total95% CIwintieloss
G1 Claude Opus vs Claude refs710.16680.129–0.2090071
G2 gpt-5.6-sol vs Codex refs710.2440—0170

From arms/device-final/aggregates/m1.json. Note the second row: it is not 0 losses-plus-zero, it is 70 losses and one order-split (win_tie_rate 0.00704). Recomputing the 150 per-order verdicts in dev-s42.codex.jsonl gives the same: 74 item-orders lost, 1 split, and the split is cw-038 — reference in order A, phone in order B.

2. So I re-ran that judge with sampling on measured

I reconstructed the published judge pipeline (judge-core.mjs: system prompt, pairwise template, blind ordering) and re-judged the four closest losses — the boundary where a count of zero is decided — using the real published pairs: app answers from df-m1-s42.jsonl, references from bench/v0.4/refs/v2/claude.jsonl (sha-verified content), 2 presentation orders × 3 repeats × 2 judge models, temperature 1.0. 48 calls, all parsed.

ItemPublished verdict (G2)gpt-5.6-sol, 3 repeatsOpus, 3 repeats
cw-038order-split (counted as tie)win · win · winloss ×3
cw-020losswin · tie · lossloss ×3
cw-011lossloss ×3loss ×3
cw-014lossloss ×3loss ×3

3. Why the win-rate was always going to read as zero measured + code

A win requires the judge to prefer the app in both presentation orders. The reference it has to beat scores an absolute 3.874 / 5 on the same rubric (abs_ref, G1 aggregates) and is written by the same model family that judges it. Under that ceiling the win condition is close to structurally unattainable: on the earlier v0.2 ruler, every configuration — including the desktop deep+thinking tier that is an order of magnitude larger — also scored exactly 0 wins. A metric that returns zero for a 35B model and for a 1.7B model is not measuring the 1.7B.

What I would publish instead. Lead with U and its interval: 0.167 [0.129–0.209], gap to the bar 0.33 — eight times the CI width, five to ten times the judge noise I measured. Then state the win-rate honestly as "≤ 4% of 71 items (95% bound); 1 item wins stably under re-judging at temperature 1.0". Same conclusion, no free ammunition for anyone who wants to dismiss the record.

4. Three more things I found while looking measured

a. The screen stage runs at α ≈ 0.25, and ~49 false positives were expected

387 registry rows resolve to about 250 independent hypotheses; 194 of them are judged screens. The frozen promotion rule (wins ≥ control+3 on 20-item slices, or Δ ≥ +0.03 at SE 0.045) implies a per-test α of about 0.25 — I checked both arms of it (P(wins≥12 | n=20, null) = 0.2517). The escalation ladder catches most of them: SEFIX-ALL gained +0.023 on the screen and was withdrawn when cross-validation gave −0.002. But of the 52 positive rows carrying a CI, Benjamini–Hochberg at q=0.05 leaves 19, and 4 of the 9 per-pack deltas (travel, health, outdoors, firstaid) do not survive a Bonferroni at k=9. The pooled Δ does not depend on them; the prose about individual packs does.

b. The "conditional prove" pack list contradicts the frozen criterion

PACKPROOF's pre-registered criterion reads sentences must use only packs that PROVE; the count was 0 of 9. The close-out report then introduces a weaker "conditional prove" rule and names 4 packs as satisfying it — a rule that appears in no criterion file — while one of those four (qa-health-se) has a pooled Δ of +0.05 with a CI of −0.15 to +0.25. That is the one place in the record where a number outruns its own rule.

c. "215 experiments without gain" is not 215 negative results

The registry splits it as 187 closed + 28 "not detected", and the registry's own definition says not-detected is not evidence of no effect; a parity arm is excluded from the count entirely. The honest phrasing is "about 250 hypotheses tested; 187 closed by their own frozen rule" — which is a stronger statement about method and a weaker one about the null.

5. The experiment this record should run next recommendation

The 4B model was killed on memory, not on quality — and the memory problem looks like a repack artefact rather than a phone limit:

Run: Qwen3-4B at Q3_K_S or IQ3_XXS, q8_0 KV, mmap-backed weights with no anonymous repack, ctx 2048, two seeds. GO iff PSS ≤ 4.7 GB, zero LMK kills, bench ratio ≥ 0.60, p50 ≤ 40 s. Falsifier: ratio < 0.56 or PSS still over budget with mmap — then the drop was correct and this audit is wrong about it. Expected: bench +0.08 to +0.12, ruler U plausibly 0.165 → 0.25–0.35.

Reproduction and limits. Bench code and items cloned from github.com/r4topunk/boar-bench; registry from boar-research; traces from the r4topunk/boar-traces dataset. Headline aggregates recomputed with the author's own m1_agg.py and matched to m1.json. My judge run used the gateway API with the reconstructed prompt, not the Codex CLI harness, and the run's frozen references file is not published (I used the repo's pinned copy, content-consistent) — so the 3/3 result on cw-038 should be read as the published judge model, sampling on, reproduced outside the original harness, not as a byte-identical rerun. Judge model id used: anthropic/claude-opus-5-5 (the one recorded in the run; the 4.6 id returns 404 on that gateway) and openai/gpt-5.6-sol. Raw verdicts: 48 rows, judgments.jsonl. Tags: verified read in code/artefacts · measured produced by this audit · recommendation untested proposal.