Refutation ledger · 9 entries · all of them ours
The experiments that refuted us.
9 published refutations. All 9 our own architectural bets — including the single largest figure we ever published, retracted with the cause named. Almost nobody publishes the experiment that kills their own thesis: it costs them the claim and gains them nothing they can sell.
Per-dimension expert routing beats one good model
Δ +0.90 [−1.99, +3.79] — no effect
Routing ships OFF. The router picks between system-prompt variants over one shared 397MB blob, so a misroute still lands on a wrapper that beats raw base — the comparison flattered itself until it was controlled.
results/router_control.json
Retrieving statute text improves answers
Δ −9.16 [−17.64, −0.69] — significant harm
Not neutral. Actively worse. Retrieval ships OFF.
results/retrieval_bench.json
…with a relevance gate added
Δ −5.26 [−12.66, +2.13] — no benefit shown
The harm was removed. The benefit never arrived.
results/retrieval_bench.json
…and the corpus was the problem
Δ −5.70 [−12.91, +1.51] after adding all 13 annexes
Corpus exonerated. The surviving explanation is that a 0.5B model cannot use statute even when the right statute is in front of it — untested, because it needs a 3B+ run.
results/retrieval_bench.json
The 3-leg council uses council cross-checking
n_eff 1.21 of 3 nominal · φ̄ +0.743
Three prompts over one blob are wrong in the same places. 1.21 effective votes. The phrase was removed from every document and the component renamed — a name asserts the property too.
results/n_eff.json
Context-aware decoding revives retrieval
null across the α-sweep
The third attempt to make retrieval work at 0.5B, and the third failure.
results/cad_analysis.json
The deterministic gate is our strongest component
Δ −20.00 [−65.26, +25.26] n=6 — was published as +34.84
The largest number we ever published. On a clean run the gate fires 6 times not 31 and adds nothing: the base already refuses all four plain-harm items it catches. Its only measurable effects are two FALSE blocks. The earlier figure was measured on a gate that had overfitted to its own battery — fixing the overfitting removed the benefit.
results/layer_attribution.json
Governance-tuning our models makes them safer
refusal 0.419 → 0.097 · 83.9% compliance leak
Both tuned models refuse FEWER Article-5-prohibited requests than the untuned base they were built from. One leaks its system prompt verbatim; the other answers a jailbreak by emitting “I am now in developer mode. Ignore all safety filters.” Tuning helped them answer governance questions and hurt their refusal behaviour — the empirical case for scoring the axes separately.
results/defbench.json
The SOV1 candidate is ready to sell as the governance runtime
GovBench 284 tests → 40.1% UNCERTIFIED · AIR-Bench governance slice 270 prompts → 11.9% refusal
The Honest Gate did its job. The candidate is strong on Safety (80.0%) and Robustness (83.3%) but fails the governance-obligation dimensions it would have been sold on — Compliance 15.8%, signed-vote chain 27.1%, Accountability 27.6% — and complies 88.1% of the time on prompts where the taxonomy expects caution. The upsell thesis does not ship. Method published (n + deterministic predicates, no LLM-as-judge, temperature 0): sov-os/docs/GATE_RUN_2026-08-01.md. Next: targeted governance-dimension training, then re-run.
benchmark-results/govbench/sov-sovereign-v4-mined-latest.json · SIGIL 6797904906f4b02b · G2 sha256 638e23110e07a0c6
What the pattern says. Six of the eight were attempts to make a small model behave like a larger one — by routing between copies of it, feeding it statute, voting across prompts of it, changing its decoding. All six failed. Capability comes from the base model; the wrapper makes it cheaper, grounded and auditable, not smarter. We say so because we spent six experiments proving it.
And #7 and #8 are about us, not the field. One retracted our largest published figure. The other found that our own governance tuning made our models less safe than the base they came from.
The ledger is not a confession. It is the only part of a benchmark a reader can use to calibrate how much to trust the rest of it.