HeurChain and LoCoMo with Recursive Improvement: The memory retrieval advantage

11 November 2026. LoCoMo DEV split plus two sealed-holdout confirmations.

Abstract

How much of a conversational assistant's accuracy comes from its memory? We answer this on LoCoMo with a memory grid. The grid holds the question set, answer prompt, evidence budget and judge fixed, and varies only what the reader is given: nothing, the Golden Prompt (only the turns LoCoMo labels as evidence), a memory system's retrieval, or the whole conversation. With no memory, gpt-6-luna, Muse Spark 1.3 and DeepSeek V4.1 Flash each answer 0 of 543 scored questions, so every correct answer depends on what memory delivers.

Reading the whole conversation is the ceiling (426 for gpt-6-luna, 495 for Muse, 492 for DeepSeek), but it costs about 117K input tokens per question.

A retrieval configuration we call S2 adds 33 to 38 correct answers for every reader against the previous HeurChain, under same-window controls. S2 uses a deeper ranked list, neighbouring dialogue turns, session openers for time questions, and compact metadata. With S2, HeurChain scores 427 with gpt-6-luna, 446 with Muse and 443 with DeepSeek, on about 5,273 input tokens per question. For gpt-6-luna it is level with reading the whole conversation, on 4.5% of the tokens. A single pre-registered run on six sealed conversations (997 scored questions) confirmed the gain: S2 beat the previous HeurChain by 68 to 18 questions where both runs of each arm agree (net +50, p < 0.0001). Most of S2's gain is on single-hop questions.

A pre-registered recursive improvement campaign then targeted multi-hop and temporal questions. It found 'dialogue windows' (configuration p04a), which present each top result together with its neighbouring turns as one gap-free stretch of dialogue. On DEV this raises gpt-6-luna to 436.3 correct against 418.7 for S2 in the same window (three runs each; multi-hop +8.3, temporal +4.3), above the full-conversation score of 426. A second pre-registered holdout run, with a stricter bar (p < 0.01), confirmed it against S2: 757.5 against 725.5 correct of 997, net +22 where both runs agree (p = 0.004), at 8% fewer input tokens.

Run in the same window as Mem0, HeurChain p04a scores 436.3 against Mem0's 384.3 with gpt-6-luna, 469 against Mem0's 404 with Muse Spark 1.3, 467.5 against Mem0's 410 with DeepSeek V4.1 Flash, using about 4,932 input tokens per question against Mem0's 8,106. The lead is mostly single-hop; on temporal questions the two are close, with Mem0 slightly ahead for gpt-6-luna.

1. Experimental conditions

Every number in this paper comes from the same harness. The table of conditions below says what was held fixed, what was varied and what was measured, so that each comparison changes one thing at a time.

Fixed (held constant across every comparison)

Independent (what we varied)

Dependent (what we measured)

Controls and statistics

2. The role of memory: the memory grid

The grid gives each of three readers five kinds of evidence, with everything else fixed. No memory scores 0 for all three readers: LoCoMo's answers are facts about these particular conversations, so a reader without memory cannot know them. The Golden Prompt, a few hundred tokens of exactly the right dialogue (the turns LoCoMo labels as evidence), already reach 382 to 438. They fall short of the full conversation mainly on temporal questions, where the turns carry relative dates ("last week") that only the session context resolves. The full conversation is the ceiling, at about 117K tokens per question. A memory system's job is to land as close to that ceiling as possible at the cost of the Golden Prompt.

Correct of 543 scored DEV questions for each evidence arm and reader (the HeurChain column is v2, before S2).
Correct of 543 scored DEV questions for each evidence arm and reader (the HeurChain column is v2, before S2).
Accuracy against input tokens per question. The large stars are HeurChain p04a and the crosses S2; Mem0 points are from the same window as p04a; the other markers are the grid arms. Each colour is one reader.
Accuracy against input tokens per question. The large stars are HeurChain p04a and the crosses S2; Mem0 points are from the same window as p04a; the other markers are the grid arms. Each colour is one reader.
ReaderNo memoryGolden PromptMem0HeurChain v2HeurChain S2HeurChain p04aFull conversationTokens/question, Mem0 / p04a
gpt-6-luna0382384.3393427436.34268,106 / 4,932
Muse Spark 1.304194044134464694958,098 / 4,887
DeepSeek V4.1 Flash0438410405443467.54928,250 / 5,028
The grid by LoCoMo category (HeurChain column is v2).
The grid by LoCoMo category (HeurChain column is v2).

3. Memory systems compared

Five memory systems were run with gpt-6-luna as reader, two runs each, under the conditions above. The two leaders were close; the others were hampered by how much of each conversation they retained or returned.

System (gpt-6-luna, two-run mean)Correct of 543Accuracy
HeurChain (v1 dates)393.572.5%
Mem038771.3%
Hindsight26548.8%
Zep (Graphiti)204.537.7%
OpenViking204.537.7%
Mean correct answers by category for the memory systems (earlier window; HeurChain before S2).
Mean correct answers by category for the memory systems (earlier window; HeurChain before S2).

3.1 Same-window head-to-head with Mem0

Mem0 was re-run in the same window as S2 and p04a, so the head-to-head below is not affected by drift between windows: three runs per arm for gpt-6-luna, two for DeepSeek (one for Mem0), and one each for Muse. Each count is a mean over runs. p04a leads Mem0 for every reader, on about 61% of Mem0's input tokens. Most of the lead is on single-hop questions. On temporal questions the two are close: p04a is ahead with DeepSeek and Muse, and with gpt-6-luna Mem0 scores 99.7 against p04a's 97 and S2's 92.7 (S2 against Mem0: 10 to 22). Temporal questions are where HeurChain has the most room to improve. The paired tests compare each question's mean correctness over runs; they are one-sided in the direction shown.

Mem0, S2 and p04a in the same window: correct answers (left) and input tokens per question (right) for each reader.
Mem0, S2 and p04a in the same window: correct answers (left) and input tokens per question (right) for each reader.
ReaderMemoryRunsMulti-hopTemporalOpen-domainSingle-hopAllInput tokens/question
gpt-6-lunaMem0359.799.78217384.38,106
gpt-6-lunaHeurChain S236292.79255418.75,272
gpt-6-lunaHeurChain p04a370.39710.3258.7436.34,932
Muse Spark 1.3Mem016799112274048,098
Muse Spark 1.3HeurChain S2174102142614515,221
Muse Spark 1.3HeurChain p04a178105182684694,887
DeepSeek V4.1 FlashMem0167104132264108,250
DeepSeek V4.1 FlashHeurChain S2269.5103.515.5257445.55,402
DeepSeek V4.1 FlashHeurChain p04a277.5107.516266.5467.55,028

Questions where p04a is right and Mem0 wrong, against the reverse (mean over runs):

ReaderMulti-hopTemporalOpen-domainSingle-hopAll
gpt-6-luna32 / 19, p = 0.0510 / 17, Mem0 ahead, p = 0.124 / 2, p = 0.3458 / 10, p < 0.0001104 / 48, p < 0.0001
Muse Spark 1.321 / 10, p = 0.0413 / 7, p = 0.137 / 0, p = 0.00846 / 5, p < 0.000187 / 22, p < 0.0001
DeepSeek V4.1 Flash24 / 14, p = 0.0711 / 8, p = 0.326 / 2, p = 0.1446 / 6, p < 0.000187 / 30, p < 0.0001

4. Closing the gap: retrieval configuration S2

S2 changes only how HeurChain retrieves and returns evidence:

Against the previous HeurChain in the same window it adds 33 (gpt-6-luna, both runs agree 38/5), 38 (DeepSeek) and 33 (Muse) correct answers, all at p < 0.0001. It sends about 600 more input tokens per question.

Previous HeurChain against S2 for each reader, with Mem0 and the full conversation for reference, and the change by category.
Previous HeurChain against S2 for each reader, with Mem0 and the full conversation for reference, and the change by category.
ReaderMulti-hopTemporalOpen-domainSingle-hopAll
gpt-6-luna+6.5+5+1+21.5+34
Muse Spark 1.3+12+5-1+17+33
DeepSeek V4.1 Flash+11+6+0+21+38

Every decision was pre-registered and taken in order. The chart below also shows the variants that did not pass: a date-resolver revision, and refinements that helped DeepSeek but not gpt-6-luna.

Every pre-registered decision on DEV, in order.
Every pre-registered decision on DEV, in order.

5. Holdout confirmation

Six conversations were sealed from the start. After the DEV work, S2 was built into the broker as options that are off by default. It was then run once on the holdout against the same broker with those options off: gpt-6-luna, two runs per arm, 997 scored questions. The pass rule was written down beforehand: a net of at least +10 at p < 0.05, with no category significantly worse. S2 averaged 721.5 correct against 670.5 and won 68 questions to 18, net +50, p < 0.0001. All six conversations improved.

Sealed holdout: change per conversation (ids redacted) and the pre-registered test by category.
Sealed holdout: change per conversation (ids redacted) and the pre-registered test by category.
CategoryQuestionsS2 better / previous betterNetp
multi-hop17610 / 2+80.0193
temporal19910 / 8+20.4073
open-domain680 / 2-21.0000
single-hop55448 / 6+42< 0.0001
all scored99768 / 18+50< 0.0001

5.2 Second holdout run: p04a against S2

After the recursive improvement campaign (section 6) promoted p04a on DEV, a second holdout run compared it with S2 on the same six conversations. It was pre-registered with a stricter bar than the first: a net of at least +10 at p < 0.01, and no category net -6 or worse at p < 0.05. S2's cells from the first run were reused unchanged, so S2's scores here (728, 723) can be compared with its first-run scores (717, 726). p04a averaged 757.5 correct against 725.5 and won 43 questions to 21, net +22, p = 0.004: PASS. Every category moved in p04a's favour, though none is significant on its own; on multi-hop plus temporal it won 20 to 9 (p = 0.03). Five of six conversations gained, one by 18 questions, and one lost 2. p04a sent 4,803 input tokens per question against S2's 5,219. Because the decision to run a second check was made after the first result was known, this is reported as a second use of the holdout, not an untouched test.

Second sealed holdout run, p04a against S2: change per conversation (ids redacted) and the pre-registered test by category.
Second sealed holdout run, p04a against S2: change per conversation (ids redacted) and the pre-registered test by category.
CategoryQuestionsp04a better / S2 betterNetp
multi-hop1769 / 5+4p = 0.21
temporal19911 / 4+7p = 0.06
open-domain683 / 1+2p = 0.31
single-hop55420 / 11+9p = 0.07
multi-hop + temporal37520 / 9+11p = 0.03
all scored99743 / 21+22p = 0.004

6. Recursive improvement on top of S2

A recursive improvement campaign then let a coding agent propose broker changes under fixed, pre-registered rules: a free evidence check, a DeepSeek screen, and promotion on three gpt-6-luna runs against the champion's three. The fitness was the gain on multi-hop plus temporal questions, with single-hop, open-domain and the total required not to fall. The first passing proposal, 'dialogue windows', folds each top result's neighbouring turns, including lower-ranked results that sit next to it, into one gap-free stretch of dialogue. On DeepSeek it scored 473 against the champion's 445 and 446. On gpt-6-luna it scored 437, 437, 435 against 425, 412, 419 (multi-hop +8.3, temporal +4.3, single-hop +3.7, total +17.7, mean of three runs each). It met every promotion rule and became the new champion. On multi-hop and temporal questions it won 39 to 22 (p = 0.02), and it gained in all four DEV conversations. The gain was larger on the half used for tuning (+11.0) than on the half held back for validation (+1.7), so some of it may be fitted to the tuning conversations. It was therefore checked on the sealed holdout (section 5.2), where it passed. Note also that S2 itself scored lower in this window (425, 412, 419) than in its first (424, 430); same-window comparison is what makes the gain readable.

7. Limits