Three benchmarks,
measured in the open.

How all three were run

Everything went through the public API, the way a customer's own integration would. Each conversation and each product was handed over to the memory endpoint into a memory of its own, dated as its dataset dates it, so none could see another's. Then every question was asked through the chat endpoint with learning switched off, so a question could not write its own answer into memory for the next question to find. Nothing used an internal path that a customer could not.

One model marks all three benchmarks, whichever model answered, so two runs are never marked by two hands. Where a benchmark's own harness names the judging prompt, that prompt is the one used: a number produced under a prompt of our own is a number comparable with nothing.

The answering model is named beside every figure, and they differ on purpose. LoCoMo was answered by the strongest rung; HERB and BEAM by one a price tier cheaper. The same HERB memory scores within about a point either way, which says the figure is the memory rather than the model.

What LoCoMo measures

LoCoMo is a public benchmark from Snap Research for long-term conversational memory. It holds ten long conversations between two people, each running to around thirty sessions spread over months, and just under two thousand questions about them: what somebody said, when something happened, what can be inferred about a person from what they have told a friend. The questions fall into five kinds. Single-hop questions ask for one fact stated once. Multi-hop questions need an answer assembled from several sessions. Temporal questions ask when, or how long ago. Open-domain questions ask what a person would likely think or do. Adversarial questions presume something nobody said, and the right answer is to decline.

It is a reasonable proxy for one half of what a company memory has to do: read a great deal of ordinary conversation once, and answer a specific question about it months later without being told where to look.

Two judges, and why only one is published

Two judges mark every LoCoMo answer, and the difference between them matters more than most people reporting on this benchmark admit. The first is the published prompt that the shared evaluation harness uses, which every result quoted in this field is produced with; it is told to be generous, and to count an answer correct when it touches the same topic as the reference. The second is ours, which tolerates no wrong fact and accepts extra detail only when it is right. The same answers score five points apart under the two, so a number quoted without naming its judge says very little. Every judged figure on this page is the shared one, because it is the only figure comparable with anybody else's. Ours is the one we develop against, and it is five points harder.

Beside them sits the benchmark's own metric. Token overlap between the answer and the reference answer, after stemming, with the adversarial questions scored by whether the answer declines in the benchmark's own words. It is harsh on an answer written in prose, and it is the only one of the three that no model's opinion enters into.

How the LoCoMo figure compares

Published results on this benchmark vary more by method than by system. The Mem0 paper (April 2025) reports judged accuracy on the answerable questions with GPT-4o-mini answering: Mem0 at 66.9, Zep at 66.0, and the whole conversation stuffed into the prompt at 72.9.

Three things to read our number against. Our answering model is named. Our judge is named, and strict. And we report the adversarial category, which several published results leave out, because an assistant that answers a question it has no record of is the failure a memory product is least allowed to have.

Why the lookup time comes from its own pass

The full run keeps dozens of questions in flight at once, which saturates the single reranker every lookup goes through: the same pipeline, on the same commit and against the same memory, reads about four times slower under the benchmark's own parallelism than at the rate a company actually asks questions. So the memory lookup time comes from a separate pass of two hundred questions asked a few at a time, and the accuracy figures come from the full run, untouched by it. A latency number taken at a rate nobody works at would describe our harness rather than the product.

What BEAM measures

BEAM ("Beyond a Million Tokens", ICLR 2026) is a benchmark of long-term memory over synthetic conversations between a person and an assistant, each long enough that the whole of it cannot sit in a prompt. Every conversation comes with twenty questions, two for each of ten abilities: recalling a fact, combining facts from several places, giving the current value after a change, noticing a contradiction, keeping to an instruction given earlier, following a stated preference, working out how long separated two events, summarising, putting events in order, and declining a question the conversation never answered.

We ran the 128K tier: twenty conversations of about 290 messages each, around three million tokens in all, and four hundred questions. Each conversation went into a memory of its own, in slices that fit the memory endpoint's reading window and dated from the conversation's own timestamps. Every question was then asked alone, with no system prompt and with memory learning off, which is how the benchmark's own baselines are asked.

How BEAM is scored, and where ours differs

Nine abilities are scored by rubric: each item of a question's rubric goes to a judge with the benchmark's own prompt, word for word, and comes back 0, 0.5 or 1. Event ordering is scored by order: each line of the answer is matched to the reference events, and the score is Kendall's tau-b between the two orders, rescaled to 0 to 1, which is the figure the benchmark's own report reads. An ability is the mean of its questions within a conversation, then across conversations, and the overall figure is the mean of the ten abilities.

The benchmark's judge is a small OpenAI model. Ours is the same Claude model that marks the other two benchmarks, so a BEAM figure here is the same prompt under a different judge: comparable in what it measures, not to the decimal. We did not put an answer to a model for event ordering where a line was blank, and we collapsed the padding in one message of 349 thousand characters that was almost entirely spaces. No other part of any conversation or question was changed.

What these numbers do not say

LoCoMo is a small sample of a narrow thing. Ten conversations: a category of ninety questions moves several points between runs for no reason. Its reference answers carry their own inconsistencies — in at least one case the transcript plainly contradicts the answer the dataset marks correct — so even a perfect system does not score a hundred on either metric. And a benchmark of two friends' conversations measures recall and attribution well while saying nothing about reading from connected systems or holding what a team agreed.

HERB is twelve products of thirty. They score anywhere from 34.6% to 67.2%, so the figure still moves as more are read — it was 62.0% on the first three and 57.1% once nine more joined them, which is the direction a flattering small sample usually goes. Clustered on the product, the interval is about four and a half points either way. It is reported here because the comparison it allows is worth making, not because the sample is settled.

None of these benchmarks connects to anything. All three hand over a fixed corpus once and then ask about it. HERB's artifacts are at least the right shape — Slack channels, specifications, meeting transcripts, pull requests — but they arrive as text, not from the systems that wrote them. What Memtro does in practice is answer from memory and from connected systems at the same time, against records that are still being written to, for people who know the business well enough to ask the awkward question. There is no public benchmark for that, so there is no number for it here.

BEAM is one tier, run once. The 128K tier is the smallest of the benchmark's tiers, and each ability rests on forty questions, so an ability moves several points between runs. Summarization and event ordering are the lowest, at 42% and 25%: both need the whole conversation in sequence rather than a single fact from it. The score is also a mean of ten abilities of very different difficulty, so it should be read beside the table, not instead of it.

All three were chosen because you can check them. That is the whole reason to run somebody else's dataset rather than one of our own: the questions, the reference answers and the scoring are theirs, the models and the judge are named, and the harness goes through the same public API a customer's integration would. A number we marked ourselves, on a corpus we picked, would be worth less than no number at all.