← Back to all research
Research · HeurChain team

The memory retrieval advantage: Recursive Improvement

Research note. A summary of the HeurChain paper: the memory grid, retrieval version S2 and its holdout check, then the recursive improvement that produced p04a. The full tables, charts and limits are in the paper.

Scope

LoCoMo DEV split: four conversations (conv-26, conv-30, conv-41, conv-50), 543 scored questions in categories 1–4. Three answer models (readers): gpt-6-luna, Muse Spark 1.3 and DeepSeek V4.1 Flash. One fixed answer prompt, one 16 KB evidence budget shared by every memory system, and one judge (gpt-6-luna with a pinned rubric) for every run.

DEV split and holdout

LoCoMo has ten long conversations. HeurChain splits them in two:

  • DEV: the four conversations above. Every experiment and every tuning

iteration runs on DEV.

  • Holdout: the other six conversations (997 scored questions; their ids are not

published). They are sealed: never used to propose, tune or choose a change. A holdout check is not an iteration. It runs only under a written, pre-registered rule with a one-time approval, scores one finalist against its baseline, and releases aggregate counts only.

The holdout has been used twice: once for S2, then once for p04a, each under its own pre-registration.

What memory adds: the memory grid

The grid holds everything fixed and changes only what the reader is given. Correct answers of 543 (earlier windows; the HeurChain column is the version before S2):

ReaderNo memoryLabelled evidence turnsHeurChain (before S2)Mem0Full conversation
gpt-6-luna0382397387426
Muse Spark 1.30419413408495
DeepSeek V4.1 Flash0438416407492
Input tokens per question (about)1104404.7K8.2K119K

With no memory every reader scores 0: LoCoMo's answers are facts about these particular conversations. The full conversation is the ceiling, at about 119K input tokens per question.

S2: a retrieval version that closes the gap

S2 is a HeurChain broker test version that changes only how evidence is retrieved and returned:

  • a ranked list of 30 results instead of 20;
  • the turns either side of the top 5 results, from the same session;
  • the opening turns of the first three sessions, for questions that ask for a time;
  • each result's metadata trimmed to its conversation id, so more evidence fits the

same 16 KB budget.

Against the previous HeurChain in the same window, with the same judge:

ReaderBefore S2S2Net, where runs agreepInput tokens per question
gpt-6-luna393427+33 (38 / 5)< 0.00014.6K → 5.3K
DeepSeek V4.1 Flash405443+38 (55 / 17)< 0.00014.8K → 5.4K
Muse Spark 1.3413446+33 (44 / 11)< 0.00014.6K → 5.2K

Change in mean correct answers, by LoCoMo category:

ReaderMulti-hopTemporalOpen-domainSingle-hopAll
gpt-6-luna+6.5+5+1+21.5+34
Muse Spark 1.3+12+5−1+17+33
DeepSeek V4.1 Flash+11+60+21+38

With S2, HeurChain scores 427 with gpt-6-luna, level with the full conversation (426) on about 5,273 input tokens per question: 4.5% of the full conversation's tokens. In the earlier-window grid, Mem0 scored 387, 408 and 407 for the three readers on about 8,100 tokens per question.

Holdout confirmation

A single pre-registered run on the six sealed conversations (997 scored questions, gpt-6-luna answering and judging, two runs per arm) compared S2 with the previous HeurChain. A question counts only when both runs of one arm are right and both runs of the other are wrong.

CategoryQuestionsS2 better / previous betterNetp
Multi-hop17610 / 2+80.019
Temporal19910 / 8+20.41
Open-domain680 / 2−21.0
Single-hop55448 / 6+42< 0.0001
All scored99768 / 18+50< 0.0001

The pre-registered test passed. All six holdout conversations gained. Most of S2's gain is on single-hop questions; temporal questions are where the most room remains.

Recursive improvement: p04a

A pre-registered recursive improvement campaign then let a coding agent propose broker changes under fixed rules, aimed at multi-hop and temporal questions, with single-hop, open-domain and the total not allowed to fall. The first passing proposal, p04a ("dialogue windows"), presents each top result together with its neighbouring turns, including lower-ranked results that sit next to it, as one gap-free stretch of dialogue.

  • On DEV, same window: gpt-6-luna scores 436.3 against 418.7 for S2 (three runs

each; multi-hop +8.3, temporal +4.3), above the full-conversation score of 426.

  • Second holdout check, with a stricter pre-registered bar (p < 0.01): 757.5 against

725.5 correct of 997, p04a ahead 43 to 21 where both runs agree (net +22, p = 0.004), on 8% fewer input tokens.

  • Against Mem0, same window: 436.3 against 384.3 (gpt-6-luna), 469 against 404

(Muse Spark 1.3) and 467.5 against 410 (DeepSeek V4.1 Flash), on about 4,932 input tokens per question against Mem0's 8,106. The lead is mostly single-hop; on temporal questions the two are close, with Mem0 slightly ahead for gpt-6-luna.

Two cautions from the paper: the gain was larger on the half of DEV used for tuning than on the half held back, and the second holdout check was decided after the first result was known, so it is reported as a second use of the holdout, not an untouched test.

Limits

  • One benchmark: LoCoMo's ten conversations are synthetic two-person chats.
  • The Mem0 and grid numbers come from earlier windows than S2; the same reader drifts

by several questions between windows, so the S2 comparison above uses a same-window control.

  • gpt-6-luna both answers and judges in its own rows. The judge is the same for every

arm, so it does not favour a memory system, but absolute scores may differ under another judge.

  • Muse numbers are single runs on DEV; both holdout checks used gpt-6-luna only.

Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (Chhikara et al., arXiv 2504.19413, April 2025) — the system HeurChain is compared against. The full HeurChain paper is HeurChain and LoCoMo with Recursive Improvement.

HeurChain paper

The full write-up behind this note, with every figure and the holdout results.

Read the paper → Download PDF
Referenced paper

This note is HeurChain's own benchmark. The linked paper is Mem0's LoCoMo paper (arXiv 2504.19413), the published system HeurChain is compared against; it is not a HeurChain paper.

Read the referenced paper →