Canonical Numbers — single source of truth for “Bento Under Test”
Author: matt.yonkovit · Re-derived from raw 2026-06-29 · Supersedes all per-section numbers in WHITE-PAPER.md Rev.2.
The rule this doc enforces: every headline number in the v2 paper cites a row here; a number changes in exactly one place (this file), then propagates. Sibling docs are regenerated, never hand-edited to drift. Where Rev.2 disagreed with the raw, the raw wins and the change is logged in §“Corrections from Rev.2.”
Provenance caveat (Threats-to-Validity): raw run records carry git_sha: "" (the harness did not
stamp a commit into records). The runs were produced by agent-samples at branch
feat/competitive-grid-harness (HEAD 572741f, the K=5 grid). Numbers below are recomputed directly
from the committed seed JSONs; the binding to a code state is the branch+commit named here, not a
per-record SHA. git_sha stamping is a tracked fix going forward.
1. Model reference (pin every identifier; drop invented precision)
| Tag in raw | Exact model | Quant | Host | Role | Notes |
|---|---|---|---|---|---|
local-133 | cyankiwi/gemma-4-26B-A4B-it | AWQ 4-bit | vLLM @ .133 | ”small” local tier | ”gemma-26B” = the A4B (active-4B MoE) build; not Google’s 27B dense. State quant. |
local-193 | research_desk baseline/recall tier | — | vLLM @ .193 | research_desk tier | GAP: exact upstream id not in MODEL-REFERENCE.md — operator to confirm before publish. |
.140 MLX fleet | Qwen3-Coder-Next (+ others) | 4-bit MLX | MLX @ .140:8080 | ”mid” tier | No public param count in SoT → Rev.2’s “Coder-Next-80B” is unsourced; drop the “80B”. |
openai:gpt-5.5 | OpenAI gpt-5.5 | — | OpenAI API | frontier tier | GAP: snapshot date not recorded. Reasoning model: no temperature, max_completion_tokens. |
Excluded by policy: thinking models unfit for multi-turn (Qwen3.5-122B etc., per MODEL-REFERENCE +
F11). All ± below are standard deviations over n seeds; 95% CI = mean ± t(n−1,.975)·sd/√n
(t=2.776 at n=5). Significance here = non-overlapping 95% CIs (a conservative, not exact, test).
2. Currency — price_book (two model tiers, both clean K=5 seed sweeps)
Gold = the price valid at that point in the update stream; executable exact grader.
| Model | arm | accuracy ± sd | 95% CI | n | source |
|---|---|---|---|---|---|
| local-133 (gemma) | baseline (no-mem, re-reads DB) | 0.950 ± 0.039 | 0.901–0.999 | 5 | evidence/matrix-133-raw/price_book/seed-baseline-* |
| local-133 | bento (write-through + supersede) | 0.923 ± 0.025 | 0.892–0.954 | 5 | seed-exp-memory-bento-* |
| local-133 | stele (raw engine) | 0.973 ± 0.037 | 0.928–1.000 | 5 | seed-exp-memory-stele-supersede-* |
| local-133 | naive cache (no invalidation) | 0.610 ± 0.148 | 0.426–0.794 | 5 | seed-exp-memory-naive-cache-* |
| gpt-5.5 | baseline | 1.000 ± 0.000 | — | 5 | results/price_book/baseline-custom/seed-* |
| gpt-5.5 | bento | 1.000 ± 0.000 | — | 5 | results/price_book/exp-memory-bento/seed-* |
| gpt-5.5 | stele | 0.997 ± 0.007 | 0.988–1.000 | 5 | seed-exp-memory-stele-supersede-* |
| gpt-5.5 | naive cache | 0.667 ± 0.122 | 0.516–0.818 | 5 | seed-exp-memory-naive-cache-* |
| gpt-5.5 | mem0-as-naive | 0.670 ± 0.127 | 0.512–0.828 | 5 | seed-exp-memory-mem0-naive-* |
Headline (significance-tested):
- bento vs naive cache: +0.313 (local-133) / +0.333 (gpt-5.5). CIs non-overlapping → significant.
- bento vs correct baseline: −0.027 (local-133), CIs overlap → statistically indistinguishable (“within noise” — now tested, not asserted). At gpt-5.5 both = 1.00.
- A caching layer without invalidation (naive, mem0-as-naive) goes stale at both tiers (~0.61–0.67); bento and stele (which supersede) stay current. This is the currency claim, isolated.
3. Cost-reuse — research_desk (multi-hop graph QA, local-193, K=5)
Static graph ⇒ no staleness ⇒ pure cost-reuse via resolved-answer recall. Executable graph-answer grader.
| arm | accuracy ± sd | wall_s ± sd | llm_calls | tokens | source |
|---|---|---|---|---|---|
| baseline (no-mem, full multi-hop each time) | 1.000 ± 0.000 | 49.3 ± 1.7 | 177 | 106,045 | results/research_desk/baseline-custom/seed-* |
| bento (recall prior answer) | 1.000 ± 0.000 | 32.3 ± 2.9 | 108 | 61,940 | exp-memory-bento-recall/seed-* |
| stele | 1.000 ± 0.000 | 32.0 ± 5.1 | 108 | 61,627 | exp-memory-stele-recall/seed-* |
| mem0 (semantic recall) | 0.940 ± 0.060 | 71.7 ± 3.8 | 285 | 106,524 | exp-memory-mem0-recall/seed-* |
| baseline-langchain | 0.960 ± 0.035 | 86.3 ± 6.1 | 166 | 103,849 | baseline-langchain/seed-* |
Headline (significance-tested):
- bento vs baseline at equal accuracy (both 1.00): wall −34.5% (49.3→32.3, CIs non-overlap → significant), llm_calls −39% (177→108), tokens −42% (106k→62k). bento ≈ stele.
- op-collapse mechanism = the llm_calls drop (177→108) — recurring questions resolve from one
recalled answer instead of re-running the multi-hop search. ⚠ Rev.2’s “3.1→2.8 turns” does NOT
exist in the raw (the
turnsaggregate field = 60 = case-count for every arm; see §7). Describe op-collapse via llm_calls, not a turns figure. - mem0 loses here: it embeds every query (semantic recall) → 285 calls, wall +45% vs baseline (71.7 vs 49.3) and lower accuracy (0.94). The per-query embedding round-trip is the cost sink.
- K=15 confirmation (small-sample rebuttal): re-run at K=15 (n tripled), the effect held — floor
acc 0.997±0.009 / 104,972 tok / 50.7±2.7s wall; bento acc 1.000±0.000 / 63,122 tok (−40%) /
32.8±2.7s wall (−35%), 95% CIs non-overlapping, effective-N=22. K=5 read −42%/−34%; K=15 reads
−40%/−35%. Source:
agent-samples results/research_desk/*/seed-{0..14}(local-193). - price_book K=15 (currency, local-133/gemma-26B) — also held: baseline 0.972±0.019 · bento
0.968±0.018 · stele 0.971±0.029 · naive cache 0.626±0.122. bento vs naive +0.342 (CIs
non-overlapping); bento vs baseline −0.004 (within noise, as at K=5’s −0.027). Both headline cells
(cost-reuse + currency) survived tripling the seeds. Backup of pre-K15 canonical:
results/_canonical_backup_20260629/.
4. Correctness-unlock — incident_rca (frontier gpt-5.5, K=5)
Unreliable/conflicting playbook corpus; agent must reach the true root despite bad docs. The ONLY
clean seed sweep for this cell is gpt-5.5 (no clean local-133 sweep exists — see §7). Executable
exact grader.
| arm | accuracy ± sd | 95% CI | wall_s | llm_calls | tokens | source |
|---|---|---|---|---|---|---|
| baseline (no-mem) | 0.460 ± 0.135 | 0.292–0.628 | 610.9 | 240 | 131,148 | results/incident_rca/baseline-custom/seed-* |
| bento (auto-inject recalled facts) | 0.777 ± 0.056 | 0.707–0.847 | 580.3 | 230 | 132,723 | exp-memory-bento/seed-* |
| bento-agent (agent-driven recall loop) | 0.893 ± 0.025 | 0.862–0.924 | 530.0 | 183 | 126,350 | exp-memory-bento-agent/seed-* |
| mem0-agent (same loop, mem0 backend) | 0.890 ± 0.030 | 0.853–0.927 | 473.6 | 259 | 126,929 | exp-memory-mem0-agent/seed-* |
| mem0-infra | 0.770 ± 0.079 | 0.672–0.868 | — | — | — | exp-memory-mem0-infra/seed-* |
| baseline-langchain | 0.490 ± 0.113 | 0.350–0.630 | 732.9 | 242 | 173,435 | baseline-langchain/seed-* |
Headline (significance-tested):
- bento vs baseline: +0.317 (0.460→0.777). CIs non-overlapping → significant correctness unlock.
- ⚠ Cost is ~FLAT at the frontier, not “−55% tokens / −39% wall” (Rev.2 §4.3): tokens 131k→133k (≈flat), wall −5%. The token-savings figures in Rev.2 were a local-133 scratch artifact, not this clean sweep — do not claim simultaneous accuracy+token wins at gpt-5.5.
- ⚠ No clean “large” sweep exists → drop Rev.2’s “grows to +0.66 at large.”
- Loop-vs-backend (the key result): bento-agent 0.893 vs mem0-agent 0.890 — Δ0.003, CIs overlap heavily → statistically indistinguishable. The win mem0-agent appeared to have over bento’s auto-inject (0.890 vs 0.777) is the agent-driven loop, not the backend: in the identical loop, bento ties mem0. This is canonical and matches Rev.2 §5 H2 (0.46→0.777).
- langchain (robust loop, no memory) is worst: 0.490, 173k tokens — below the custom baseline.
5. Memory TYPES (this session’s measured A/Bs — §4.4–4.7)
| § | metric | bento | comparator | source |
|---|---|---|---|---|
| 4.4 graph retrieval (isolation, 6-q paraphrased, ×3) | needle-in-context | 1.00 | vector 0.17, bm25 0.00, naive 0.17 | shape-bench/graph-vs-vector/* |
| 4.4 NDCG@10 (g) — single chain, NO surface cue (n=6×3) ⭐ ranking | 0.78 (local) | vector 0.11, bm25 0.07 | graph ≫ all when no lexical bridge. NOTE: pg-raggraph naive=vector+BM25 (not pure-vector); global is the strongest graph mode at scale. ndcg-* | |
| 4.4 NDCG@10 (g) — scaled 8-chain, conventional (named, n=16) | 0.72 (global) | bm25 0.54, vector 0.40 | graph (global) beats vector+bm25 on realistic multi-hop. ndcg-scaled-* | |
| 4.4 NDCG@10 (g) — scaled 8-chain, pure-lexical codename (n=16) | 0.79 (global) | bm25 0.89, vector 0.67 | bm25 WINS the pure-lexical key (fusion-weight config knob dilutes it — not a missing-BM25 bug; BM25 is in every bento mode). Reported, not hidden. | |
| 4.4 multi-model synthesis (retriever fixed) | answer acc | 26B 0.33 → 31B 0.50 → gpt-5.5 1.00 | — (retrieval model-independent) | shape-bench/graph-vs-vector/multimodel-synth-* |
| 4.5 code chunking (254 QA, 66 files) | functions kept intact | 1.00 | fixed-1200 0.49 / fixed-2000 0.60 | shape-bench/code-aware/chunking-ab-* |
| 4.5 | top-1 = complete fn | 0.83 | fixed-size ~0.35 | (boundary alignment, not size) |
| 4.5 code retrieval — MRR / NDCG@10 (complete-fn) ⭐ ranking | MRR 0.896 / NDCG 0.919 | fixed-size MRR 0.384 / NDCG 0.398 | region-MRR 0.896 vs 0.687 = the fragmentation tax | |
| 4.6 KB currency (9 q, 3 generations) | post-update correct | 1.00 (a17) | naive cache 0.33 | a17/destructive-delete stack. On a18 append-only, superseded versions are retained → chunk-string metric reads 0.33 (artifact); answer-level + rank-1 currency stay 1.00. shape-bench/evolving/kb-currency-*, §7.5 |
| 4.6 time-travel | as_of-exclude ✓ / replay ✓ (a18) | full | replay UNBLOCKED by pg-raggraph v0.5.0a18 (#90 append-only re-ingest, evolution_tier=structural): between-version as_of returns the historical version (chunk + answer). shape-bench/evolving/asof-replay-*, §7.5 | |
| 4.7 procedural reuse (4 ep × 6 orders, ×3 models) | refused ops | 3 (−75%) | no-memory 12 | shape-bench/procedural/reuse-* · reproduced on a18/.140 Qwen-Coder: 2 vs 8 (−75%), see §7.5 |
| 4.7 | accuracy | 1.00 | no-memory 1.00 (tie) | tokens ~flat (v2) / +7% on the Qwen-Coder re-run — win is failed-action elimination |
6. Spend (single figure everywhere)
Paid LLM (gpt-5.5) total: $77.2 / $200 budget. (Rev.2 cited “$72.73” in §5, “$77.02” in handoff, “$8 of $500” in model-tier-comparison — all reconciled to $77.2 / $200; this session’s two new gpt-5.5 cells added ≈$0.20.) Local GPU tiers (.133/.140/.193) = $0.
7. Corrections from Rev.2 (what changed and why — the honesty log)
| Rev.2 claim | Canonical reality | Disposition |
|---|---|---|
| §4.2 “op-collapse fired (3.1→2.8 turns)“ | turns=60 for every arm (case-count, not per-task turns) | Remove the number; use llm_calls 177→108 (−39%) as the mechanism |
| §4.3 incident_rca “0.50→0.89 (+0.39) med, −55% tok, +0.66 large” | clean sweep = gpt-5.5 only: 0.46→0.777 (+0.317), tokens ~flat; no clean large sweep | Re-anchor to gpt-5.5 sweep; drop −55% tok + +0.66 large (local-133 scratch, not canonical) |
| §5 gpt-5.5 rca “0.46→0.777” vs model-tier-comparison “0.583→0.833” | seed sweep = 0.460→0.777 | §5 was right; model-tier-comparison.md is stale → fix sibling |
| §4.1 price_book “0.92/0.95” vs §5 “0.880/0.923” | §4.1 = local-133 (0.923/0.950); §5 = gpt-5.5 (1.00/1.00) | Label by model tier; both canonical, different stacks |
| Abstract “frontier tier pending” | gpt-5.5 grid ran; spend $77.2 | Remove “pending”; state spend |
| §4.6 time-travel “FIXED/verified” | as_of-exclude works; version replay blocked (pg-raggraph#90) | Re-scoped to exclude-only earlier this session; #90 RESOLVED 2026-06-29 by pg-raggraph v0.5.0a18 — replay now ✓ (see §7.5) |
| “tight CIs” over n=3–5 | those were std; n=5 (med), pre-reg said K≥10 | Relabel std; add 95% CIs (above); disclose K-shortfall |
| ”executable graders, not LLM judge” | true for memory cells; §4.4 graph cell is LLM-judged (26B + must-include) | Scope the claim; carve out the graph cell |
Pre-registration shortfall (disclose): PRE-REGISTRATION.md committed to K≥10 seeds; every
reported cell is K=5 (research_desk-large K=2 not used in canonical). Stated openly: effect sizes
on the headline deltas (+0.31 currency, +0.32 correctness, −34% wall) are large enough to clear n=5
CIs, but the K≥10 target was not met — a real limitation, not hidden.
7.5 Tool-sync re-measurement — stele 0.6.10 + pg-raggraph a18 + .140 Qwen3-Coder-Next (2026-06-29)
The two “not delivered” gaps the v2 paper flagged were closed/advanced by a tool sync, then
re-measured live. Version delta: these cells move from the v2 SUT (stele 0.6.9 / pg-raggraph a17 /
.133 gemma-4-26B distill) to stele 0.6.10 / pg-raggraph a18 / .140 MLX Qwen3-Coder-Next — so
TWO variables change (tool version AND distill/ask model). Read them as a new measurement, not a diff
against the v2 numbers. evolution_tier=structural (live since 2026-06-17). Regression suite 11/11 green
on the new stack.
Gap B — full version replay: NOW DELIVERED (was ✗). pg-raggraph a18 #90 makes a same-id
re-ingest append-only when evolution_tier!=off (closes the prior version’s effective_to, opens the
new effective_from, writes a document_versions edge). Proven live (asof-replay-*.json,
all-replay-pass=True): a 30→90→7-day same-id re-ingest retains 3 version rows; an as_of inside
each window returns ONLY that historical version at both chunk and /v1/ask answer level
(v1→30, v2→90, current→7).
- Side-effect on §4.6 currency metric: append-only RETAINS superseded versions (down-ranked by the
supersession penalty; current ranked #1). The chunk-string “no stale lingering” metric
(
kb_currency_test.py) therefore reads 0.33 — an artifact of retention, not a regression (the v2 “1.00” was partly an artifact of destructive deletion). Honest currency = answer-level (/v1/ask→current) or rank-1: both 1.00 post-a18.
Gap A — self-evolving currency: measured, model-bounded. stele 0.6.10 #88 (asserted event-date
supersession) + #89 (schema-blind column guard — closes the F1 root cause the F13/F16/F18 PR flagged)
advance the primitives. Re-measured staleness (currency_fix_experiment.py, 10 evolving-fact scenarios,
distill = .140 Qwen-Coder):
| arm | stale↓ | clean↑ | recall_top_current↑ |
|---|---|---|---|
| baseline | 1.00 | 0.00 | 1.00 |
| consolidate_0.82 | 1.00 | 0.00 | 1.00 |
| recall_adaptive | 1.00 | 0.00 | 0.40 |
- Store retirement fails (
stale=1.0/clean=0.0) on every arm — even the consolidate pass can’t retire siblings. The supersession slot keys on(subject_id, aspect)and Qwen3-Coder-Next is a coder model emitting unstable subject/aspect → old/new land in different slots. Distill-model artifact, not a stele-version regression (compose docs: “the distill model dominates cross-session staleness… too weak for real currency”); a gemma-class temp=0 distill retires cleanly (v2 used.133gemma-4-26B). - Recall stays current under the recency strategy (
recall_top_current=1.0), drops to 0.4 underadaptive. From the agent’s view (top recall hit) continuity recall is current; the cost is store hygiene + strategy sensitivity.
§4.7 procedural reuse — reproduces. .140 Qwen-Coder (4 ep × 4 orders): failed ships 2 vs 8
(−75%), accuracy tie (1.00), model-calls flat, tokens +7% (carrying the recalled procedure). Win =
failed-action elimination, as in v2. shape-bench/procedural/reuse-local-140-qwen-*.
7.6 letta comparison (2026-06-30) — closes the v2 “letta not run” gap
We stood up a self-hosted letta server (Docker, v1.12.1) and tested it two ways, mirroring the mem0
loop-vs-backend split. Chat on .140 MLX (local) and OpenAI (frontier); embeddings via ollama
nomic-embed-text. Adapter: agent-samples/memory/letta/.
- A — archival backend: letta archival memory behind the same write/recall interface as the bento / mem0 adapters.
- B — full agent: letta’s self-editing MemGPT agent (core-memory edits).
| claim | letta-A (archival backend) | letta-B (full agent) | bento ref |
|---|---|---|---|
| C1 currency (post-update correct) | 0.00 (append-only; can’t supersede) | 1.00 (gemma-4-31B AND gpt-5) | bento 0.968 / stele ~0.97 |
| C7 procedural reuse (failed ops) | 2 vs 8 = −75% (ties bento) | — | bento −75% |
The split is the finding: letta’s archival backend recalls fine but cannot supersede — 0.00 on currency (worse than a naive cache; dedup/consolidation is open letta feature-request #3116), yet it ties bento on procedural reuse (−75%), because that task is write-once/recall-many and needs no supersession. letta’s agent stays current (1.00) on any capable model, matching bento/stele.
Capability map — what letta structurally cannot do (these are “only bento plays,” not “bento wins”):
- C4 graph retrieval — letta has no graph RAG.
- C5 code-aware chunking — letta has no code-aware chunker.
- C6 time-travel / version replay — letta has no point-in-time replay.
Two model-compatibility limitations (letta’s, not bento’s):
- letta’s agent cannot be driven by a coder model:
Qwen3-Coder-Nextloops ~50 tool-calls/turn (~82s) with no final answer. A capable instruct model (gemma-4-31B, gpt-5) drives it cleanly in ~3–5s. - letta v1.12.1 cannot drive gpt-5.5: it sends
reasoning.effort='minimal', which gpt-5.5 rejects (supported: none/low/medium/high/xhigh). We usedgpt-5as the frontier proxy (1.00, same as gemma).
Evidence: shape-bench/evolving/letta-currency-*.json, shape-bench/procedural/reuse-local-140-qwen-*.json.
Cost-reuse (C2) and correctness (C3) — not benchmarked, by finding (not by omission):
- C2 cost-reuse — blocked by letta’s recall API. The resolved-answer cache the other backends use gates
a hit on cosine similarity (≥ 0.6) so a novel question never gets a wrong cached answer. letta’s recall
API (
passages.list/passages.search, SDK v1.12.1) exposes no similarity score — results carry onlycount/index— so a relevance-gated cache cannot be built on its backend; only its agent could judge relevance. A naive clone would inject the nearest passage for every query (false positives), measuring the API limitation, not cost-reuse. Recorded as a finding, not a number. - C3 correctness — disproportionate build, agent path already characterized. incident_rca’s arms are agent-driven; a letta arm means letta’s full agent driving RCA with custom tools (not an adapter swap). The agent path is already shown by the currency cell (letta-B = 1.00 on a capable model). Deferred.
7.7 SC-005 pull-path currency — gemma distill validates clean retirement (2026-06-30)
The §7.5 “model-bounded” caveat is now closed with a measurement. Re-ran currency_fix_experiment.py
(same 10 evolving-fact scenarios) with ONLY the stele distill endpoint switched from the coder model to
gemma — STELE_DISTILL_LLM_BASE_URL=http://192.168.1.140:8081/v1 (.140 :8081 = gemma-4-31B-it-qat;
:8080 stays Qwen-Coder; these are single-resident MLX servers that ignore the model field, so the
base URL must move), stele-shim recreated. The graph/RAG SUT (PGRG_LLM_MODEL/OPENAI_BASE_URL:8080)
was left unchanged — only the distill path moved.
| distill | arm | stale↓ | clean↑ | recall_top_current↑ |
|---|---|---|---|---|
| Qwen-Coder (:8080) | baseline | 1.00 | 0.00 | 1.00 |
| gemma-4-31B (:8081) | baseline | 0.10 | 0.50 | 0.60 |
| gemma-4-31B | consolidate_0.82 | 0.10 | 0.50 | 0.60 |
| gemma-4-31B | consolidate_0.70 | 0.00 | 0.60 | 0.60 |
| gemma-4-31B | maxwin_1 | 0.10 | 0.50 | 0.60 |
| gemma-4-31B | recall_adaptive | 0.10 | 0.50 | 0.50 |
- Confirms the §7.5 hypothesis directly: the distill model dominates currency. A gemma-class temp=0 instruct model takes stale-sibling retention 1.0 → 0.1 (baseline); the consolidate sweep at 0.70 retires the residual → 0.0. Clean supersession rises 0.0 → 0.5–0.6.
- Honest tail: ~0.3 of scenarios end “only-old active” (gemma over-retracts/mislabels a minority);
recall_top_currentis 0.6 (the coder model’s 1.0 was high only because BOTH siblings stayed active and recency surfaced the newer one).recall_adaptivedoes not help — use the recency strategy. - What this means for the loop: the failure-recovery loop’s PUSH path is deterministic and
model-independent (
collapse_superseded, bento#893); the PULL path (stele recall returning active-only) needs a capable temp=0 distill, and gemma delivers it. Validated, REVERTIBLE via two.envlines. Evidence:sc005-currency/SC-005-pull-path-currency-gemma.md.
7.8 Failure-recovery loop — W1 net savings (SC-007, 2026-06-30)
The Tier-1 failure-recovery loop (bento#893) is built (detect → EV-gate → redact → store → push, 27 tests
green) and runnable (python -m failure_learner). W1 cold→warm net measured by failure_learner.w1
(scripted: deterministic token costs via tiktoken o200k + measured latency, N=12, bootstrap 95% CI):
| quantity | value |
|---|---|
| wasted-call tokens avoided (failed cmd + error) — lower bound | 43 |
| pushed-line context cost (per session) | 32 |
| net tokens per prevented call (lower bound) | +11 |
| wasted-call latency avoided | 27.0 ms (95% CI 25.6–28.3) |
| learner overhead (one offline scan, amortized) | 0.54 s |
- Latency saved per prevented call is the unambiguous win — warm sessions skip the doomed call entirely; an in-prompt correction line adds ~0 latency. 27 ms on a fast local fixture; slow calls (network, builds, large reads) save far more.
- Token net is a LOWER BOUND — it counts only what re-enters context deterministically (the failed call + its error result) and EXCLUDES the dominant real saving, the model’s doomed-turn generation (that needs the live LLM path). Even so it is +11 on a one-line error.
- Safety invariant: drop any correction whose net ≤ 0 (never ship one that costs more context than the
call it prevents). Real-data run currently surfaces 0 corrections (no recurring cross-dir failures in the
sample); surfacing engages as recurrence ≥2 accumulates. Evidence:
w1-failure-loop/w1-fixture-*.{json,md}.
8. Scratch / non-canonical files (quarantine — do NOT cite)
price_book/{bento-k5.json 0.96, baseline-k5.json, pb-stable*.json 0.20–0.73, pb-final.json 0.978, *-smoke.json}; incident_rca/{rca-*, base-*, *-smoke} (local-133 exploratory, inconsistent);
research_desk/{rd-*.json}. These are dev iterations at varied sizes/stacks; the seed-*.json K=5
sweeps named in §2–4 are canonical. A README in each evidence/.../ dir will name canonical vs
archive (WP-011).