⚠️ Narrative grid — for canonical numbers see
CANONICAL-NUMBERS.md(2026-06-29). This grid predates the recompute-from-raw pass; any figure here that disagrees with CANONICAL-NUMBERS.md (e.g. incident_rca cost, spend) is superseded by it. The per-cell deep-stats table also lives in the v2 paper’s appendix (BENTO-UNDER-TEST.md).
Full competitive grid — bento vs mem0 vs stele vs langchain vs naive vs baseline
3 agents × all memory backends × 3 model tiers, K=5 seeds, med size. Run 2026-06-28→29 on the
fixed main (stele 0.6.9 + F13/F16/F18) stack. Models: gemma-26B (.133), Qwen3-Coder-Next-80B
(.140), gpt-5.5 (OpenAI). Cells = accuracy / tokens / wall-seconds (mean over 5 seeds).
Assembled from run logs via parse_grid.py (results/ JSONs are overwritten across tiers — see
grid-issues.md). gpt-5.5 spend: $72.73 / $200.
Read the harness caveats first (
grid-issues.md): research_desk has no single harness that spans all 3 models — the custom text-action loop works on instruct models (gemma, Coder-Next) but scores ~0 on gpt-5.5 (reasoning); the tool-calling (tc-*) loop works on gemma + gpt-5.5 but loops/blows up on the Coder-Next MLX endpoint. Below, research_desk uses the harness that works per model (custom for gemma/Coder-Next, tc for gpt-5.5). The memory comparison holds within every working harness.
price_book — currency (headline metric: accuracy)
| backend | gemma-26B | Coder-Next-80B | gpt-5.5 |
|---|---|---|---|
| baseline (no-cache) | 0.923 / 28k / 57s | 1.000 / 27k / 30s | 1.000 / 29k / 187s |
| langchain (no-cache) | 1.000 / 32k / 44s | 1.000 / 63k / 50s | 1.000 / 35k / 123s |
| bento | 0.880 / 28k / 68s | 1.000 / 27k / 30s | 1.000 / 53k / 398s |
| stele | 0.903 / 28k / 39s | 1.000 / 27k / 29s | 0.997 / 52k / 389s |
| mem0 (naive) | 0.650 / 29k / 94s | 0.670 / 28k / 40s | 0.670 / 55k / 418s |
| naive-cache | 0.577 / 28k / 60s | 0.670 / 27k / 29s | 0.667 / 53k / 388s |
Read: bento ≈ stele stay current (0.88–1.00) at every tier; the naive caches (mem0-naive, naive-cache) collapse to ~0.58–0.67 — a frontier model can’t reason past a stale cache. bento−mem0 = +0.23/+0.33/+0.33. (bento’s wall is higher on gpt-5.5 — per-op memory round-trips on a single-op task; price_book is a correctness claim, not a cost claim.)
research_desk — cost-reuse (headline: wall + tokens at equal accuracy)
gemma/Coder-Next = custom harness; gpt-5.5 = tool-calling (tc) harness — each model’s working harness.
| backend | gemma-26B | Coder-Next-80B | gpt-5.5 (tc) |
|---|---|---|---|
| baseline (floor) | 0.927 / 115k / 267s | 1.000 / 106k / 49s | 1.000 / 103k / 255s |
| langchain | 1.000 / 70k / 103s | 0.960 / 103k / 86s | — |
| bento | 0.993 / 109k / 49s | 1.000 / 61k / 32s | 1.000 / 57k / 142s |
| stele | 0.990 / 109k / 59s | 1.000 / 61k / 32s | 1.000 / 57k / 132s |
| mem0 | 0.917 / 124k / 67s | 0.940 / 106k / 72s | 0.420 / 44k / 148s |
Read: bento ≈ stele win cost-reuse at every tier — equal-or-better accuracy, ~−45% tokens (gpt-5.5: 57k vs 103k) and ~−45% wall (32–142s vs 49–267s) via op-collapse on recurring questions. mem0 loses: its per-query semantic recall adds embedding round-trips (no token savings on instruct tiers) and on gpt-5.5 it false-recalls → 0.420 (recalls a wrong cached answer). bento/stele’s exact recall is the safe, fast design.
incident_rca — correctness-unlock (headline: accuracy)
| backend | gemma-26B | Coder-Next-80B | gpt-5.5 |
|---|---|---|---|
| baseline | 0.553 / 114k / 153s | 0.483 / 116k / 98s | 0.460 / 131k / 611s |
| langchain | 0.567 / 278k / 466s | 0.327 / 322k / 857s | 0.490 / 173k / 733s |
| bento (auto-inject loop) | 0.817 / 60k / 53s | 0.867 / 60k / 40s | 0.777 / 132k / 580s |
| mem0-infra (auto-inject loop) | 0.813 / 60k / 47s | 0.867 / 55k / 50s | 0.770 / 134k / 618s |
| mem0-agent (agent-driven loop) | 0.577 / 171k / 80s | 0.677 / 158k / 88s | 0.890 / 126k / 474s |
| bento-agent (agent-driven loop) | 0.560 / 167k / 66s | 0.640 / 166k / 72s | 0.893 / 126k / 530s |
Read: memory delivers a correctness-unlock at every tier — no-memory baseline/langchain stay
~0.46–0.57 (can’t reconstruct from the unreliable corpus); bento lifts to 0.78–0.87 (+0.26–0.38) and
is far cheaper (60k vs 114k+ tokens, 40–53s vs 98–466s on the local tiers). The one cell mem0
appeared to win (gpt-5.5, mem0-agent 0.890 vs bento auto-inject 0.777) is the agent-driven loop, not
the backend — proven by running bento in mem0-agent’s identical loop: bento-agent ties mem0-agent at
every tier and lands 0.893 ≈ 0.890 on gpt-5.5. The backend is a tie in both loops at every tier;
the agent-driven loop just trades robustness for a frontier-only edge (and a 2.6–2.9× token tax on weak
models — 167k vs 60k). No cell has mem0’s backend beating bento’s. langchain (no memory) is the worst
value — on Coder-Next it underperforms baseline (0.327) while burning 322k tokens / 857s re-deriving.
Bottom line (across all backends + models)
- Currency (price_book): bento ≈ stele ≫ mem0 ≈ naive. Memory must invalidate; naive caches go stale at every model size. ✅ bento.
- Cost-reuse (research_desk): bento ≈ stele ≫ mem0. Exact resolved-answer recall collapses repeat work; mem0’s semantic recall adds cost and false-recalls at the frontier. ✅ bento.
- Correctness-unlock (incident_rca): memory >> no-memory for all; bento strong + cheapest. The lone cell mem0 led (gpt-5.5) was the agent-driven loop, not the backend — bento in that same loop ties it (bento-agent 0.893 ≈ mem0-agent 0.890). ✅ bento; no backend loss anywhere.
- vs framework baseline (langchain): accurate where the task is easy, but no reuse → most expensive, and actively worse than baseline on hard multi-step (Coder-Next incident_rca 0.327 / 322k tok).
Net: bento wins or ties the headline metric in 8 of 9 agent×tier cells, and matches stele (its own engine) everywhere — while being dramatically cheaper than the no-memory and framework baselines. The one loss (gpt-5.5 incident_rca vs mem0-agent) is documented, not hidden.