Research note. Every number below is real, quoted from HeurChain's internal benchmark report (updated 2026-11-08) with its scope caveats intact. The referenced paper is Mem0's published LoCoMo paper, the result this benchmark is compared against.
Scope
Four LoCoMo DEV conversations (conv-26, conv-30, conv-41, conv-50), 543 scored questions in categories 1–4. The 158 adversarial category-5 questions were answered but not judged and are excluded from every accuracy number here. Each system ran twice independently (runs a and b). OpenViking was added as a sixth system on 2026-11-08.
DEV split and holdout
LoCoMo has ten long conversations. HeurChain splits them in two:
- DEV — four conversations (conv-26, conv-30, conv-41, conv-50; 543 scored
questions). Every experiment, comparison and tuning iteration runs on DEV, including the automated improvement loop (RSI, recursive self-improvement). An iteration is one proposed change scored on DEV.
- Holdout — the other six conversations (997 scored questions; their ids
are not published). They are sealed: never used to propose, tune or choose a change. A holdout check is not an iteration. It runs only under a written, pre-registered rule with a one-time approval, scores one finalist against its baseline, and releases aggregate counts only, so the result is not tuned to the questions it is measured on.
The holdout check for HeurChain broker test version S2 is in the full Six systems on LoCoMo DEV report: S2 beat the previous HeurChain by a net 50 questions (p < 0.0001).
Every number in this note is a DEV result from an earlier window. The report also compares S2 with Mem0 in the same window.
Headline (mean of two runs)
| Run | Correct / 543 | Accuracy |
|---|---|---|
| HeurChain + date writing — mean | 393.5 | 72.5% |
| Mem0 — mean | 387.0 | 71.3% |
| HeurChain — mean | 383.5 | 70.6% |
| Hindsight — mean | 265.0 | 48.8% |
| Zep (Graphiti) — mean | 204.5 | 37.7% |
| OpenViking — mean | 204.5 | 37.7% |
What the numbers actually say
- The top-three gap is small, not a win. HeurChain + date writing leads Mem0 by 6.5 questions, from a different run window. All three sit within 10 questions of each other, while run-to-run flips are 34–43 questions per system. On this DEV split the ranking of the top three is not separable from run noise.
- Mem0 still leads temporal questions; date writing closes about half the gap. Mem0 scores 98.5 of 122 temporal questions against 89.5 with date writing and 80.0 for plain HeurChain (same-window control 82.5).
- HeurChain leads single-hop by 17 questions (235.0–235.5 of 287 vs Mem0's 218.0). Date writing leaves single-hop and multi-hop essentially unchanged on the four-run replication.
- Zep and Hindsight abstain far more under these matched settings (UNKNOWN on 47.9% and 37.6% of scored questions, vs 16.9% HeurChain and 18.0% Mem0). Neither is a fair reading of those systems as published.
- OpenViking ties Zep at 204.5, abstaining 52.1% of the time; its published 80–83% comes from agent integrations with repeated search and archived-session reads, so it is not comparable.
Cost note
Cost per run tracks prompt caching, not accuracy: HeurChain $0.26, Mem0 $0.45, Zep $0.48, Hindsight $0.30 — but one HeurChain run cost $0.44 and the other $0.07 for identical prompts. Compare systems on token volume, not dollars.
Related paper
Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (Chhikara et al., arXiv 2504.19413, April 2025) reports 66.9% for Mem0 and 68.4% with graph memory on all ten LoCoMo conversations with a gpt-4o-mini judge — a different split, judge, and answer model, so the comparison is directional only.