Bento Under Test

Bento is an open-source agent-memory and retrieval system — deployable self-hosted or as a managed service — built as a single FastAPI backend-for-frontend (BFF) that composes four retrieval/memory ingredients behind one ingredient registry. Clients talk only to the BFF; it routes each capability to the ingredient that owns it. Three ingredients run as HTTP services over a shared substrate (PostgreSQL + Valkey); lede runs in-process because a network hop would dwarf its sub-millisecond work. Being open source, the system under test is itself inspectable and clonable — the reproducibility argument extends from the harness to the product.

/v1/ask resolves kb-{id} and queries chunks; it has no path into stele. Recall resolves str(kb_id) and reads stele; no path into chunks. So the currency/accumulation claims (C1/C3/C7, stele) and the retrieval claims (C4/C6, pg-raggraph) genuinely test different machinery — not two views of one store.

• Ingest — /v1/ingest → chunkshop picks a strategy (symbol_aware for code, sentence-aware for prose) → pg-raggraph-proxy writes documents + chunks and, when graph is on, extracts the entity graph; effective_from is stamped for point-in-time (F13).
• Ask — /v1/ask → resolve kb-{id} → pg-raggraph retrieval (naive / hybrid / local graph modes) → the models service synthesizes a grounded, cited answer.
• Memory — POST /v1/memory writes a structured fact to stele; an optional replaces atomically supersedes the prior value (the currency mechanism, C1). /v1/recall returns active-only memories — superseded rows are never surfaced.

Every memory arm runs against a no-memory control on the identical task, model, and grader. The design can disprove each claim.

A mutable price book; a hot SKU's price is updated mid-stream; the agent must always answer the current price. A no-memory baseline re-reads the database (correct, but redoes work). A cache that never invalidates should go stale.

Against the correct re-read baseline, bento is −0.027 at the local tier with overlapping 95% CIs — statistically indistinguishable. That is the honest meaning of "tracks the baseline": equal correctness, with a reusable cache instead of a full re-read.

Multi-hop graph QA over a static dependency graph, with recurring questions. A resolved-answer cache can collapse a repeated question's multi-hop work into one recalled answer. Static graph, so no staleness — pure cost. Both memory arms reach accuracy 1.00, so the comparison is cost at equal correctness.

Root-cause analysis where the document corpus is unreliable and conflicting. Memory accumulates verified findings and recalls them on recurring incident classes. Bars below show accuracy with 95% confidence-interval whiskers.

A 6-document relational chain buried in distractors; questions paraphrased so the answer entities never appear in them and keyword/vector search has no surface to match.

Holding the retriever fixed and varying only the answer model shows retrieval is solved and synthesis is model-bound: answer accuracy climbs 0.33 → 0.50 → 1.00 across the 26B, 31B, and frontier models on the same graph-retrieved context.

Honest boundary: on easy questions that name the chain entities, bm25 also succeeds — graph's edge is specifically the multi-hop, no-surface-overlap case.

bento's symbol-aware chunker vs LangChain's fixed-size splitter, over the same 66 files / 254 functions, same embedder. Metric: is the complete target function the top-1 hit?

Code-aware chunking keeps 100% of functions whole (vs 49–60%) and returns the complete function as the top hit 2.4× more often. The mechanism is boundary alignment, not chunk size: a larger fixed window barely changes completeness because windows still cut across function edges.

fixed-size finds the right region reasonably (region-MRR 0.687) but the complete-function unit ranks far lower (MRR 0.384) — that 0.687→0.384 gap is the fragmentation tax in ranking terms: it locates where the function is, but no single chunk holds the whole thing.

A task where some orders' ship is refused until validate_address is called — a recovery procedure the agent discovers by reading the error. Over recurring episodes, no-memory re-discovers the failure every time; bento stores the procedure once and recalls it. Identical result across all three model tiers.

A 75% reduction in repeated failed operations at equal accuracy. Token cost is flat to slightly higher (the recalled hint costs tokens), so this is a win where failed operations carry real cost — side effects, money, rate limits, irreversible actions — and cost-neutral where retries are free.

1. Harness heterogeneity. research_desk uses a text-action loop on instruct models and tool-calling on the frontier; win counts are claimed only within a model's working harness, never pooled.
2. Sample size. n=5 seeds, below the pre-registered K≥10; small deltas are under-powered.
3. Stack snapshot. The headline cells ran on one point release (stele 0.6.9 / pg-raggraph a17 + F13/F16/F18). The memory-type cells (C6, C7) were re-confirmed on a second stack (stele 0.6.10 / a18), giving a partial cross-version check. The local-tier and frontier cells are different tiers, labeled rather than pooled.
4. Cross-box mem0 in one cell. The price_book mem0 figure ran cross-box with infer=False; the fair mem0 comparison is the same-box, identical-loop decomposition in C3, where the competitive claim rests.
5. One LLM-judged cell. C4's answer accuracy uses the local 26B as judge; its retrieval metric is deterministic. "Executable graders" is not generalized to this cell.
6. git_sha not stamped. Raw records carry empty git_sha; numbers bind to the named harness branch/commit, not a per-record SHA. Tracked for the public release.
7. Identifier gaps. Two model identifiers (local-193 upstream id, gpt-5.5 snapshot) are not recorded — flagged, not invented.

This version re-anchors every number to recomputed raw data and removes claims the raw did not support:

• The "3.1→2.8 turns" mechanism (not in the data — turns is the case count; replaced by model-call counts, 177→108).
• "−55% tokens / +0.66 at large" on incident_rca (exploratory local run; replaced by the +0.317 frontier result at flat cost).
• "Frontier tier pending" in the abstract (the frontier ran; spend is stated as $77.2 / $200).
• Time-travel "FIXED" (re-scoped to currency-yes / replay-no, pg-raggraph#90) — #90 since resolved by pg-raggraph a18 (2026-06-29): replay now delivered, §7.5.
• "Tight CIs" over n=5 (relabeled as standard deviations, with real 95% CIs and a disclosed K-shortfall).

The full mapping is in evidence/CANONICAL-NUMBERS.md §7.

1.2 (2026-06-30): the autonomous failure-recovery loop moved from pre-implementation to built + measured (C7; W1 net in §7.8); cross-session store currency quantified against the distill model (C6, §7.7); a per-section "So what" reading added.

On controlled tasks with executable graders and a no-memory control, bento delivers measurable value on the axes a hosted agent-memory layer should: it keeps answers current under updates, reduces cost on recurring work at equal accuracy, and unlocks correctness the base model cannot reach on an unreliable corpus. Its distinctive surface — graph retrieval, code-aware chunking, supersession-backed currency, procedural reuse — produces measured benefits over external comparators. The one cell a competitor led decomposes to the agent loop, not the backend; in the identical loop, bento ties.

We are equally explicit about what is not claimed: token savings at the frontier on the correctness task (cost is flat there, not reduced). The autonomous recovery loop, unproven in the v2 body, is now built and measured at Tier-1 — a prevented call saves ~27 ms at a net-positive token cost (C7, §7.8); the Tier-2 analyzer is deferred. (Two more follow-ups since the v2 body: pg-raggraph a18 delivered full version replay, §7.5; and a letta comparison ran — its agent ties bento on currency and procedural reuse, its archival backend can't supersede, and it has no graph, code-aware, or time-travel surface, §7.6.) The pre-registered seed count was not met (K=5 vs K≥10).

The complete breakdown a stats-literate reviewer wants: accuracy mean ± standard deviation, 95% CI, seed count, wall time, model calls, and tokens per cell — all recomputed from the committed raw seed records. Source of truth: evidence/CANONICAL-NUMBERS.md; narrative grid: evidence/full-competitive-grid.md.

Notes: C4/C5/C6 are deterministic (no seed variance — the NDCG runs show sd=0.000 across 3 repeats); C7 is identical across all three model tiers (the reuse benefit is behavioral, not model-bound). C7's token cost is flat-to-slightly-higher — the win is the −75% in refused operations, not tokens. Significance reads (95% CI overlap) are in §5; threats to validity in §6.