Research note. A summary of the HeurChain paper: the memory grid, retrieval version S2 and its holdout check, then the recursive improvement that produced p04a. The full tables, charts and limits are in the paper.
Scope
LoCoMo DEV split: four conversations (conv-26, conv-30, conv-41, conv-50), 543 scored questions in categories 1–4. Three answer models (readers): gpt-6-luna, Muse Spark 1.3 and DeepSeek V4.1 Flash. One fixed answer prompt, one 16 KB evidence budget shared by every memory system, and one judge (gpt-6-luna with a pinned rubric) for every run.
DEV split and holdout
LoCoMo has ten long conversations. HeurChain splits them in two:
- DEV: the four conversations above. Every experiment and every tuning
iteration runs on DEV.
- Holdout: the other six conversations (997 scored questions; their ids are not
published). They are sealed: never used to propose, tune or choose a change. A holdout check is not an iteration. It runs only under a written, pre-registered rule with a one-time approval, scores one finalist against its baseline, and releases aggregate counts only.
The holdout has been used twice: once for S2, then once for p04a, each under its own pre-registration.
What memory adds: the memory grid
The grid holds everything fixed and changes only what the reader is given. Correct answers of 543 (earlier windows; the HeurChain column is the version before S2):
| Reader | No memory | Labelled evidence turns | HeurChain (before S2) | Mem0 | Full conversation |
|---|---|---|---|---|---|
| gpt-6-luna | 0 | 382 | 397 | 387 | 426 |
| Muse Spark 1.3 | 0 | 419 | 413 | 408 | 495 |
| DeepSeek V4.1 Flash | 0 | 438 | 416 | 407 | 492 |
| Input tokens per question (about) | 110 | 440 | 4.7K | 8.2K | 119K |
With no memory every reader scores 0: LoCoMo's answers are facts about these particular conversations. The full conversation is the ceiling, at about 119K input tokens per question.
S2: a retrieval version that closes the gap
S2 is a HeurChain broker test version that changes only how evidence is retrieved and returned:
- a ranked list of 30 results instead of 20;
- the turns either side of the top 5 results, from the same session;
- the opening turns of the first three sessions, for questions that ask for a time;
- each result's metadata trimmed to its conversation id, so more evidence fits the
same 16 KB budget.
Against the previous HeurChain in the same window, with the same judge:
| Reader | Before S2 | S2 | Net, where runs agree | p | Input tokens per question |
|---|---|---|---|---|---|
| gpt-6-luna | 393 | 427 | +33 (38 / 5) | < 0.0001 | 4.6K → 5.3K |
| DeepSeek V4.1 Flash | 405 | 443 | +38 (55 / 17) | < 0.0001 | 4.8K → 5.4K |
| Muse Spark 1.3 | 413 | 446 | +33 (44 / 11) | < 0.0001 | 4.6K → 5.2K |
Change in mean correct answers, by LoCoMo category:
| Reader | Multi-hop | Temporal | Open-domain | Single-hop | All |
|---|---|---|---|---|---|
| gpt-6-luna | +6.5 | +5 | +1 | +21.5 | +34 |
| Muse Spark 1.3 | +12 | +5 | −1 | +17 | +33 |
| DeepSeek V4.1 Flash | +11 | +6 | 0 | +21 | +38 |
With S2, HeurChain scores 427 with gpt-6-luna, level with the full conversation (426) on about 5,273 input tokens per question: 4.5% of the full conversation's tokens. In the earlier-window grid, Mem0 scored 387, 408 and 407 for the three readers on about 8,100 tokens per question.
Holdout confirmation
A single pre-registered run on the six sealed conversations (997 scored questions, gpt-6-luna answering and judging, two runs per arm) compared S2 with the previous HeurChain. A question counts only when both runs of one arm are right and both runs of the other are wrong.
| Category | Questions | S2 better / previous better | Net | p |
|---|---|---|---|---|
| Multi-hop | 176 | 10 / 2 | +8 | 0.019 |
| Temporal | 199 | 10 / 8 | +2 | 0.41 |
| Open-domain | 68 | 0 / 2 | −2 | 1.0 |
| Single-hop | 554 | 48 / 6 | +42 | < 0.0001 |
| All scored | 997 | 68 / 18 | +50 | < 0.0001 |
The pre-registered test passed. All six holdout conversations gained. Most of S2's gain is on single-hop questions; temporal questions are where the most room remains.
Recursive improvement: p04a
A pre-registered recursive improvement campaign then let a coding agent propose broker changes under fixed rules, aimed at multi-hop and temporal questions, with single-hop, open-domain and the total not allowed to fall. The first passing proposal, p04a ("dialogue windows"), presents each top result together with its neighbouring turns, including lower-ranked results that sit next to it, as one gap-free stretch of dialogue.
- On DEV, same window: gpt-6-luna scores 436.3 against 418.7 for S2 (three runs
each; multi-hop +8.3, temporal +4.3), above the full-conversation score of 426.
- Second holdout check, with a stricter pre-registered bar (p < 0.01): 757.5 against
725.5 correct of 997, p04a ahead 43 to 21 where both runs agree (net +22, p = 0.004), on 8% fewer input tokens.
- Against Mem0, same window: 436.3 against 384.3 (gpt-6-luna), 469 against 404
(Muse Spark 1.3) and 467.5 against 410 (DeepSeek V4.1 Flash), on about 4,932 input tokens per question against Mem0's 8,106. The lead is mostly single-hop; on temporal questions the two are close, with Mem0 slightly ahead for gpt-6-luna.
Two cautions from the paper: the gain was larger on the half of DEV used for tuning than on the half held back, and the second holdout check was decided after the first result was known, so it is reported as a second use of the holdout, not an untouched test.
Limits
- One benchmark: LoCoMo's ten conversations are synthetic two-person chats.
- The Mem0 and grid numbers come from earlier windows than S2; the same reader drifts
by several questions between windows, so the S2 comparison above uses a same-window control.
- gpt-6-luna both answers and judges in its own rows. The judge is the same for every
arm, so it does not favour a memory system, but absolute scores may differ under another judge.
- Muse numbers are single runs on DEV; both holdout checks used gpt-6-luna only.
Related paper
Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (Chhikara et al., arXiv 2504.19413, April 2025) — the system HeurChain is compared against. The full HeurChain paper is HeurChain and LoCoMo with Recursive Improvement.