Six things we mistake for memory

When an agent answers correctly about its own past, at least six different mechanisms could have produced that answer. Only one of them is memory.

This is the whole problem. The behaviours are identical from the outside — fluent, confident, specific, often moving. The text is always flawless; that is what these models are for. And every instrument the field currently uses to evaluate agent memory reads the text.

• Retrieval The answer is produced because the information is there — in a document, a database, a context window — and is fetched. This is the only one of the six that most people mean when they say an agent remembers.
• Reasoning The answer is derived. Given the shape of the question and a little context, a sufficiently good model can work out what the answer must have been, with no record of any kind. Our Check 2 is entirely about this: five of seven models solved a memory probe at 62–79% with an empty context. They were not remembering. They were solving.
• Semantic plausibility The answer is the one that sounds most likely. Not derived, not retrieved — simply the highest-prior continuation. If the true answer and the plausible answer coincide, which they usually do, no test built on free text can separate them. This is why our third probe forces a choice between two alternatives that are symmetric by construction: when the truth is a coin, plausibility has nothing to grip.
• Conditioning The answer follows the content of the document in context, whatever that content is. Give the model a true past and it answers truly; give it a fabricated one and it answers falsely, with identical conviction. This is what we measured: seven models out of seven, no hesitation, no abstention.
• Autobiographical continuity The claim that something persists across episodes and accumulates — that today's answer depends on having lived the intervening days, not merely on holding a summary of them. This is the interesting hypothesis. It is also the one nobody has a validated instrument for, which is the finding this project reports.
• Simulated identity The performance of being someone with a past: first-person reference, ownership, consistency of voice, the texture of a self. Real models do this beautifully whether or not there is anything behind it. Our counterfeit control scores a fabricated life and a lived one at +0.032 apart on identity markers — statistically equivalent. The performance is real. It is just not evidence.

We do not claim that a language model "has" memories in any human sense. We make no claim about inner life, experience, or consciousness, and nothing in this project bears on those questions.

Instead we compare three documents placed in a model's context, and ask whether behaviour attributed to memory depends on the historical veridicality of their content:

• Veridical memory document Corresponds to the agent's actually recorded history.
• Counterfeit memory document A fabricated life that never happened, matched for length and format.
• Binding-corrupted memory document The agent's real episodes with the bindings swapped — the right topics attributed to the wrong interlocutor.

And then the only question this project asks:

The question is not "does the agent remember?" — but: does your instrument respond to the difference between these three?

It is a smaller question than "is this thing conscious", and it has the singular advantage of being answerable. It is also the question you have to answer first, because until your instrument can separate a lived past from a well-made forgery, every number it produces is a number about performance.

For the three studies in which we asked it, and what came back: