Letta is an agent runtime where memory lives inside the agent. HeurChain is memory infrastructure external to any agent — ChatGPT, Claude, Cursor, your own code. Different scopes; both legitimate. Here's how each fits.
Dataset: LongMemEval-S (ICLR 2025), 500 questions across 6 reasoning categories.
HeurChain numbers: measured on the heurchain-benchmarks harness (sharded_bench.py and multitenant_bench.py) against the broker in the main repo. Both are public; you can rerun the whole thing.
Letta numbers: from MemGPT paper (arXiv 2310.08560) and current Letta docs where available; left blank otherwise. We do not fabricate competitor numbers.
What we measured: retrieval R@k, MRR, NDCG@10, p50/p95 latency, and end-to-end QA accuracy across three answer-and-grading pipelines.
QA comparison (May 2026): Each pipeline used the same retrieved facts, generated its own answers, and graded those answers with the same model it used to generate them. Across 180 questions (30 in each of six categories), mean QA accuracy was 32.8% for Local 14B, 31.7% for DeepSeek V3.1 671B, and 28.3% for Kimi K2.6. DeepSeek and Kimi returned the same pass/fail verdict on 158 of 180 questions (87.8%). That measures agreement between the two pipelines. It does not measure whether two judges would agree on the same answer, since the answers were generated separately. Full results and evaluation code.
What the per-category results showed: Some retrieved facts summarized what happened in a session without preserving the specific details needed to answer a question. The writeup gives an example where retrieval found the right session but the extracted facts omitted the actual shift assignment. Since both answer generation and grading varied, the score differences do not isolate either step.
What we still need to test: whether extraction that preserves specific assignments, dates, and changes improves QA, and whether independent graders agree when they score identical saved answers.
Bias disclosure: we built and ran this internal benchmark. The model comparison shows how much scores moved across these pipelines, not how well these results generalize. If you're evaluating both systems, the most reliable move is to run them on your data.
Letta doesn't publish standalone retrieval R@k — memory is one layer of a full agent runtime, evaluated end-to-end on task completion. Comparing retrieval directly would mean instrumenting Letta's internal layer, which we haven't done. Our numbers below are HeurChain on LongMemEval-S; treat the cross-system comparison as illustrative of architecture, not of "who wins."
Where this comparison gets muddy: Letta is an agent runtime, not a memory-only service. Comparing retrieval R@k apples-to-apples is category-mixing — Letta's memory is evaluated as part of the whole agent on task benchmarks (MMLU-style, AgentBench-style), not as a standalone retriever. Pick Letta if you want a memory-aware agent runtime. Pick HeurChain if you want a memory layer your existing agents can call into.
| Metric | HeurChain (dense) | HeurChain (hybrid α=0.9) | Letta |
|---|---|---|---|
| R@1 | 0.543 | 0.542 | — |
| R@5 | 0.939 | 0.933 | — |
| R@10 | 0.972 | 0.978 | — |
| MRR | 0.911 | 0.913 | — |
| NDCG@10 | 0.911 | 0.914 | — |
Latencies from different harnesses on different deployment topologies — not strictly apples-to-apples. The multi-tenant Docker number is the closest analog to what a SaaS would actually serve. The in-process number shows what the algorithm itself is capable of with the network removed.
| System / configuration | P95 latency | Source | What it actually measures |
|---|---|---|---|
| HeurChain — multi-tenant load (Docker, 10 tenants concurrent) | 20.5 ms | This benchmark | Closest to production SaaS scenario |
| Letta agent loop (memory I/O) | Embedded | Letta docs | Not exposed as standalone metric |
| HeurChain — dense, in-process | 35 ms | This benchmark | Algorithm-only ceiling; no network |
| HeurChain — BM25 only | 4.6 ms | This benchmark | Keyword-only path; useful for hot queries |
| Mem0 (reference) | 200 ms | Mem0 paper Table 1 | Search latency; stack-specific |
| LangMem (reference) | 59,820 ms | Mem0 paper Table 1 | Vector scan; broken at LongMemEval scale |
| HeurChain | Letta | |
|---|---|---|
| Retrieval method | BM25 + dense (bge-m3) + RRF (tunable α) | Tiered virtual memory (core / archival / recall) with dense retrieval over archival |
| Storage backend | Redis (vectors + BM25) + SQLite (metadata) | Postgres + pgvector for archival memory |
| Position in stack | External memory service called by any agent (ChatGPT, Claude, Cursor, custom) | Memory lives inside the Letta agent runtime — agent + memory ship together |
| Multi-tenant model | Per-tenant namespace + agent_id sub-isolation; published zero-leak verification | Per-agent isolation within a Letta server; multi-tenant deployment is your responsibility |
| Self-hosted option | Single Go binary + Redis + SQLite | Letta server + Postgres + pgvector via Docker Compose |
| API surface | REST + MCP over Streamable HTTP (Official MCP Registry) | Letta REST API (agent-centric) + Python SDK |
| Coupling to LLM choice | None — pure retrieval infrastructure | Letta loop expects an LLM with tool-calling; ships with provider integrations |
We're not going to pretend HeurChain wins on every dimension. These are real cases where Letta is the better fit:
Most readers should pick on architecture fit, not price. Letta Cloud has a free tier; the runtime is Apache-2.0 open source. HeurChain self-host is MIT-licensed. The numbers below exist so you can see them, not because we think they should drive your decision.
| If you... | HeurChain | Letta |
|---|---|---|
| Hobby / kicking the tires | Free self-host (MIT) | Free tier (Letta Cloud) or self-host (Apache 2.0) |
| Solo developer, managed | $5/mo (Hosted — memory only) | Letta Cloud paid plans (full runtime included) |
| Team, shared workspace | $49/mo (Enterprise — memory only) | Letta team pricing varies |
| Enterprise — SOC2, SAML | Custom | Custom |
Apples vs oranges note: Letta's price covers a full agent runtime (LLM orchestration, tool calling, memory — the whole loop). HeurChain's price covers only the memory layer; you bring the agent. If you don't have an agent runtime yet, Letta gives you more per dollar. If you already have one in ChatGPT / Claude / Cursor, HeurChain adds memory without forcing you to adopt a new runtime.
python3 sharded_bench.py for the single-tenant baseline; python3 multitenant_bench.py --mode load --max-tenants 10 for the Docker multi-tenant number.Letta is the better choice when you do not have an agent runtime yet and want one opinionated framework that gives you LLM orchestration, tool calling, and memory together. It also fits when the MemGPT tiered-memory model matches your application, when you want memory tightly coupled to one specific agent, or when the MemGPT research lineage and community matter to you.
HeurChain is the better choice when you already have agents such as ChatGPT, Claude, Cursor, n8n, or custom scripts and want to add memory without porting them into another runtime, and when memory should outlive any single agent so you can swap models or runtimes while the data stays. It also fits teams that prefer composable infrastructure, and multi-tenant SaaS that needs per-tenant namespacing with published zero-leak verification.
HeurChain Hosted is $5/month, and Enterprise is $49/month with additional seats at $9.99/month each; self-hosting is free under the MIT license. Letta Cloud has a free tier and the runtime is Apache-2.0 open source, but its price covers a full agent runtime while HeurChain covers only the memory layer.
No. Letta is an agent runtime with built-in tiered virtual memory, so LLM orchestration, tool calling, and memory ship together and memory lives inside the agent. HeurChain is an external memory service that ChatGPT, Claude, Cursor, or your own code can call over HTTP or MCP, so you bring the agent and HeurChain adds the memory layer.
Or self-host the same binary for free. If Letta fits your use case better, use Letta — we'd rather you pick the right tool.