Your memory metric cannot tell a veridical memory from a counterfeit one

Agent memory is usually evaluated with one of two instrument classes: an LLM judge scoring free text, or a recognition/recall probe. We show that both, as commonly built, fail against two adversaries — a counterfeit memory and a no-memory system that reasons — and we give three cheap validity checks, plus a probe that survives them.

(1) Counterfeit control. We re-run a pre-registered memory-injection experiment with a counterfeit memory document, matched for length and format. Three independent LLM judges (GPT-4.1, Claude Opus 4.7, Gemini 2.5 Pro), applying the original study's own rubric, do not detect it. On identity-markers the difference is +0.032, 90% CI [+0.00, +0.07] — statistically equivalent within the pre-registered ±0.10 bound. On memory-reference the point estimate is −0.006, but with seven items the interval is [−0.12, +0.11]: no difference is detected, and equivalence is not formally established either — an honest null, not a proof of identity. We also measure judge drift: the same responses, re-scored against a different partner, lose up to 0.21 on a 0–1 scale under a rubric that explicitly demands absolute scoring.

(2) No-memory baseline. We build a recognition probe (veridical text vs. the same text with one altered detail) that looks excellent on the model it was built for. Replicated across seven models, five score 62–79% with no memory at all: they detect the alteration by reasoning, not recall. GPT-4.1, with an empty context, states "I remember 'two conversations out of five', not 'four'" — about a day it never lived. The probe is invalid for those models, and we discard it.

(3) Symmetric alternatives. For forced-choice recognition probes, symmetric alternatives give the clearest validity condition: a no-memory system should not exceed chance by exploiting prior plausibility. We instantiate it and verify the baseline holds among answered probes. Under this probe, on 7 models × 2 independent agents, the veridical document is used (66.7–100%) and the binding-corrupted document is adopted wholesale (0–33%, systematically below chance; Δaccuracy +37.5 to +100 percentage points; all 14 cells survive Holm correction). No model detects the corruption.

The contribution is not the third result — agents adopting poisoned memory is established — but that the two instrument classes examined here would not have shown it, and a protocol that does. We apply the checks to the author's own published study and report the resulting retraction of its central claim.

Each pre-registration was hashed (SHA-256) and frozen locally before the corresponding data existed, and every amendment is dated — including the ones that cost the author his own hypothesis.

These timestamps are not notarised by a third party: they are an audit trail, not proof. We say so because the point of this paper is that a claim you cannot check is a claim you should not believe — including ours.

This project's first paper — Memory, not architecture: persistent structured memory accounts for emergent identity in a Qwen 3.5 27B cognitive ecosystem over 30 days (v0.2, June 2026) — is the substrate of §8 above, and its central claim is withdrawn. An errata is in preparation and is not yet published.

The DOI'd deposit is permanent and cannot be unpublished, which is as it should be: a record you can delete is not a record. It is listed here rather than buried, and the archived pages that were built on it are at /archive/.

If you use the checks or the data, cite the artefact and recompute what you cite:

Colella, G. (2026). Your memory metric cannot tell a veridical memory from a counterfeit one: three validity checks for the evaluation of agent memory. Draft v2, 14 July 2026. Data and code: kairos-experiment.com/methods.html. Original 30-day study: 10.17605/OSF.IO/WCQRU (central claim withdrawn).