Publishing the LongMemEval results for VENETA WorldModel's memory and self-improvement layer

Figure. LongMemEval-S, all 500 questions, writer Claude Opus 5.5, the benchmark's official GPT-4o judge, three runs on each side. The bar is the mean of three and the thin line is their range. Below, the on arm split by ability; the number beside each name is how many questions that ability has. The same values are in the table above.

Why we built our own exam

VENETA WorldModel’s memory and self-improvement layer was built for on-premise operations: mission-critical control rooms, places where the data cannot leave the building, where the model writing the answer is the customer’s own local LLM.

No memory benchmark measured that. The existing ones all assume an external frontier model does the remembering, and on-premise that assumption does not hold. They also ask how well a system recalls a conversation the benchmark wrote itself. There is no operator in it — nobody approving, rejecting or correcting. So you cannot ask the questions that matter: “What did we approve last time, and did the gate hold?” “Does a rule we rejected once come back in different words?” In mission-critical operations, that is exactly what memory is for.

So we built one. The benchmark and its results are on the benchmark page.

But we were curious about the opposite case

What happens if the layer is used for general purposes rather than on-premise — an ordinary user, with a frontier model as the writer? That is in fact the situation the memory products on the market assume.

So we swapped the writer for Claude Opus 5.5 and measured with LongMemEval, the representative general-purpose memory benchmark. Not our exam: we did not design it, and it does not assume our environment.

What LongMemEval is trying to measure

Its goal is whether a chat assistant remembers old conversations. Each question arrives with roughly 50 past sessions of that user’s history — 489,731 characters on average, about three novels. The answer hides in one utterance somewhere inside.

In human terms: you mentioned something in passing six months ago, and now you are asked about it. Five abilities are tested:

Information extraction — what did you say at the time?
Multi-session reasoning — answers that only appear when pieces from several conversations are put together
Knowledge update — you moved house and changed jobs. Does it know the current answer?
Temporal reasoning — “what did you say before you did that?”
Abstention — asked about something that was never in the conversation, does it say it does not know? That one is harder than it sounds

How we measured

1. A fresh memory for every question. That question’s past sessions go in as dated records. No model is used at write time — nothing is summarised or extracted, the rows go in as they are. No memory carries between questions, so nothing learned on one question helps another.

2. Three ways of finding it. What is close in meaning, what shares words, what names the same thing — the three results are fused into one ranking. The local model also rewrites the question six different ways as search terms, and each found utterance brings one turn either side with it. What that gathers stays under 60,000 characters, which is about 12 % of the 489,731.

3. The answer is written from that block alone. The writer never sees the original conversation — only what we selected.

4. The control, with the layer off. The same writer, no memory layer, the conversation simply truncated to 60,000 characters and pasted in — the same budget as the block on the other side, and run three times as well.

5. Somebody else grades it. The benchmark’s official GPT-4o judge compares against the answer key. Not our scorer. As a cross-check we had a local model grade the same answers. Across 6 runs of the two arms the two judges reached the same verdict on 490 to 498 of the 500 questions. The figures we publish are the official judge’s.

6. Three runs. So that one lucky run never becomes a published number, every cell is run three times and reported as a mean with its range. No question selection — all 500.

7. Errors count. If the model breaks format and the answer comes back empty, that counts as wrong. We do not re-run it. It happened once in this measurement, and that one is inside the number below.

The result

	official GPT-4o judge
memory layer on (3 runs)	94.5 % [93.6–95.2]
memory layer off (3 runs)	19.7 % [19.2–20.2]

By ability:

ability	questions	correct, per run	mean of 3 runs
knowledge update	78	76 / 77 / 77	98.3 [97.4–98.7]
preference within a session	30	30 / 30 / 30	100.0
what the assistant said	56	55 / 55 / 54	97.6 [96.4–98.2]
what the user said	70	66 / 69 / 68	96.7 [94.3–98.6]
temporal reasoning	133	123 / 125 / 124	93.2 [92.5–94.0]
multi-session reasoning	133	118 / 120 / 120	89.7 [88.7–90.2]

Abstention is not a seventh row — it is inside four of the rows above. The opening said this benchmark also asks whether the assistant says “I don’t know” when the answer is not in the conversation, and that is why no row carries it. Those questions number 30, and they are not a type of their own: they are drawn from what the user said 6 · multi-session 12 · temporal reasoning 6 · knowledge update 6. So the question counts above add to 500 and the 30 are already inside that 500 — do not add them on. Taken on their own, 29 / 28 / 27 were right, 93.3 [90.0–96.7].

The 100.0 is a 30-question ability, not a claim. “Preference within a session” has 30 questions in this benchmark, and all 30 were right in each of the three runs. It is a small denominator and should not be read as evidence of perfection — which is why this table prints the question count and the per-run counts beside every mean. The hardest cell, multi-session reasoning, is 89.7 over 133 questions, and that is the one we are working on.

The conditions these numbers carry. Writer Claude Opus 5.5 · all 500 questions · official GPT-4o judge · three runs on both sides, mean with range · the same 60,000-character budget for both arms — the cap on the recall block with the layer, the length of the tail kept without it · a fresh memory per question · no model at write time.

The two arms are at the same budget. The cap is 60,000 characters on both sides, and without the layer every question filled it — not one conversation in this benchmark is shorter than that. With the layer the block measures 60,945 characters on average, 1.6 % longer because of its headings.

And the product’s default is different from both. These figures come from a 60,000-character block, sized to a haystack three novels long. The product’s default recall budget is 3,000 characters, sized to an operator’s decision record. Different task, different block.

There is one cell where the control scores full marks

Taken on its own, the arm with the layer off is perfect at abstention — 30 / 30 / 30 of 30, 100.0 in all three runs. With the layer on it is 93.3.

This is not good news. The control sees nothing but the last 60,000 characters of the transcript, so it answers “I don’t know” to almost everything. On these questions “I don’t know” is the answer. The arm that knows nothing scores full marks when abstention is read on its own.

Where that full score sits in the total makes it plain. Of the 99 / 96 / 101 answers the control gets right, 30 are these questions, every run. Over the other 470 it gets 69 / 66 / 71 right — 14.6 % [14.0–15.1]. That is where the 19.7 % comes from.

So why is the layer-on arm at 93.3? Not because it invented anything. We opened all 6 of its misses across the three runs, and every one of them named the absence first, correctly.

Asked how many times the user had baked egg tarts in the past fortnight, it answered: “I checked every conversation from 2023-05-16 to 2023-05-30, and egg tarts don’t come up anywhere.” Had it stopped there it would have been marked right. It did not stop — it went on to say what the user had actually baked in that period, a chocolate cake. All true, and the better answer for an operator, but this benchmark’s abstention scoring wants the answer to end at the absence.

Not all of it is the scoring, though. In 3 of the six it named the absence and then attached a figure anyway — estimating a saving against a bus fare that was never in the record, for instance. Putting a number on a premise that is not there is simply wrong.

Those three were also invited by our own exam paper. The instructions we give the writer include “when the excerpts give part of what is needed, answer from that part, say what is missing, and still state a number, date or name on the last line”. The bus fare is exactly that shape — the taxi fare was in the record, so this was a “part of it is there” case, the last line was required to carry a figure, and the model did as it was told.

So we built a change that switches that instruction off only when the premise is absent from the record, and measured it. Abstention came out right every time — but on questions that could be answered the model began saying “there is no record of that” when there was, and the set lost more than abstention gained. We did not adopt it. The rule we had written down before measuring said a loss on answerable questions blocks adoption, and a rule is not edited after the numbers arrive.

That instruction exists only in the harness that runs the benchmark, not in the product. So those three answers are not how the product behaves day to day; they are something our way of measuring produced. Our own exam paper cost us points, and writing that down is better than leaving it out.

So abstention is not a number to read on its own. Knowing nothing earns full marks; knowing a great deal costs you points for saying what you know. Read the two scores apart and they point in opposite directions.

What we do not claim from this

We do not say we are better than other memory products. There are no competitors’ scores on this page. We have not run a head-to-head under matched conditions, so we are not entitled to the comparison.

We do not say one benchmark measures memory. LongMemEval looks at long conversations between a person and an assistant. It does not ask whether the record of what a person approved, rejected and corrected in a control room survives. That is what our own benchmark asks, and it is on the benchmark page.

We do not say 94.5 % is a product-configuration number. As written above, the block size differs.

Related notes: Benchmark · Choosing a local writer · Tool-call reliability