Research note. HeurChain's internal miss-taxonomy analysis (2026-11-07). Every claim below is tagged in the source report as verified (computed over the frozen evidence cells) or inferred; counts are quoted with that tagging. The referenced paper is Mem0's published LoCoMo paper, the system this analysis is compared against.
Scope
LoCoMo DEV split: conv-26/30/41/50, 543 scored questions, faithful matched replay of HeurChain evidence, two runs per system, retrieval frozen per system.
DEV split and holdout
LoCoMo has ten long conversations. HeurChain splits them in two:
- DEV — four conversations (conv-26, conv-30, conv-41, conv-50; 543 scored
questions). Every experiment, comparison and tuning iteration runs on DEV, including the automated improvement loop (RSI, recursive self-improvement). An iteration is one proposed change scored on DEV.
- Holdout — the other six conversations (997 scored questions; their ids
are not published). They are sealed: never used to propose, tune or choose a change. A holdout check is not an iteration. It runs only under a written, pre-registered rule with a one-time approval, scores one finalist against its baseline, and releases aggregate counts only, so the result is not tuned to the questions it is measured on.
The holdout has been used twice, each time for a HeurChain broker test version: S2 beat the previous HeurChain by a net 50 questions (p < 0.0001), then p04a beat S2 by a net 22 (p = 0.004). Both checks are in the HeurChain paper.
Every count in this note comes from DEV. The fixes this analysis pointed to were later built and checked on the holdout; see the paper.
Headline findings
- Temporal is an answer-side problem on evidence HeurChain already retrieves. Of 49 temporal misses, 36 had all labelled evidence turns packed into the prompt; zero misses were packing misses. In 31 of the 36, the gold turn contains a relative time expression ("yesterday", "last year") that the reader must resolve against the session header — and HeurChain either abstained or copied the phrase unresolved.
- Multi-hop is a retrieval-coverage problem. Of 55 multi-hop misses, 45 are retrieval-side (30 partial, 15 none) and only 10 are answer-side. The HeurChain–Mem0 multi-hop gap (59.0 vs 62.0 of 106) sits inside run-to-run noise (HeurChain flips 16 multi-hop questions between its own two runs).
- The single-hop lead is the thing to protect. HeurChain leads 50 single-hop questions to Mem0's 27 behind; in 24 of the 50, Mem0's paraphrased memories dropped the specific detail that raw turns keep. Any change that replaces raw turns with extracted facts risks this lead.
- HeurChain's temporal answers are unstable. 14 flipped temporal verdicts between runs vs 3 for Mem0 — the noise floor any temporal experiment must beat, and itself a symptom of hedged prompts on the abstain/answer boundary.
Implied levers
Temporal and multi-hop need different fixes: write resolved dates inline into the evidence text for temporal (headers alone are read but not resolved), improve retrieval coverage for multi-hop — and hold every change against a single-hop non-regression constraint with replicate runs.
Related paper
Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (Chhikara et al., arXiv 2504.19413, April 2025) — the system this taxonomy uses as its reference point on temporal and multi-hop questions.