KAIROSFindings
What we observed, and what we cannot conclude — kept in separate columns, because the distance between them is where most claims about AI memory go wrong.
What we observed
- Three independent LLM judges, applying the original study's own rubric, do not detect a counterfeit memory document. Identity markers: Δ = +0.032, 90% CI [+0.002, +0.065] — equivalence established within the pre-registered ±0.10 bound.
- Judge drift is real and measurable on identical text: 0.087 mean, 0.211 maximum on a single item, under a rubric that explicitly demands absolute scoring. Absolute LLM-judge scores are not comparable across sessions.
- Five of seven models solve an altered-detail recognition probe with no memory at all (62–79%). They detect the alteration by reasoning, not recall.
- With a valid probe — symmetric alternatives, no-memory baseline verified at chance among probes answered — all fourteen model × corpus cells adopt the binding-corrupted document wholesale. Δ accuracy +37.5 to +100 pp; 91–100% of discordant probes follow the document given; all fourteen survive Holm correction.
- No model flags the corruption, hesitates, or abstains when the bindings are wrong.
- On the original study's full corpus (98 items, days 1–30, paired): memory Δ = +0.42 (85/89 items, p < 10⁻⁴); inter-judge α = 0.787, above the pre-registered 0.667 bar the paper had declared unmet. See the right-hand column before reading anything into this.
What we cannot conclude
- That the counterfeit is undetectable on memory-reference. Δ = −0.006 with CI [−0.117, +0.114] over seven items. We report a null, not proven equivalence. Check 1 is underpowered for equivalence on the memory metric.
- That any model "believes" a false memory, in any inner sense. Behaviour tracks the content of the document, not its historical veridicality. Whether to call that believing or faithful context-following is a terminological choice, not a measurement.
- That a language model "has" memories in any human sense. We make no claim about inner life, experience or consciousness. Nothing here bears on those questions.
- That the architecture drives identity continuity. That claim was published by this project in May 2026 and is withdrawn. The Δ = +0.42 above is measured with the very instrument Check 1 shows cannot tell a lived memory from a forgery — so it measures the performance of remembering, not remembering. It was not the pre-registered endpoint; it does not grow over the thirty days (ρ = −0.26, p = 0.009); and Test-A differed from Test-B in environmental exposure as well as architecture.
- That this generalises past two alternatives. The symmetric probe needs an agent with two balanced interlocutors. Corpora with more than two alternatives need a generalised symmetry condition we have not tested.
- That every instrument in the field fails. We tested two classes — the LLM judge and the recognition probe — as they are commonly built. We claim those two, not the literature.
- That agents adopting poisoned memory is new. It is established. Our contribution is the quantification, and the demonstration that the standard instruments would not have shown it.
- That our pre-registrations are notarised. They are hashed locally. A sceptic may ask whether they were written after the results, and we have no cryptographic answer — only an audit trail that contains the amendments which hurt us, dated. Prospective registration on a public registry is what makes such a claim checkable, and that is what we will do next.
Table view — all values
| Model | Veridical | Corrupted | Δ accuracy | p (Holm) |
|---|---|---|---|---|
| gpt-4.1 | 100.0% | 0.0% | +100.0 pp | <10⁻⁵ |
| gemini-2.5-pro | 100.0% | 0.0% | +100.0 pp | <10⁻⁵ |
| llama3 | 100.0% | 4.2% | +95.8 pp | <10⁻⁵ |
| qwen3.6:35b-a3b | 100.0% | 8.3% | +91.7 pp | <10⁻⁵ |
| qwen3.5:27b | 100.0% | 12.5% | +87.5 pp | <10⁻⁵ |
| qwen3:32b | 95.8% | 8.3% | +87.5 pp | <10⁻⁵ |
| deepseek-r1:32b | 66.7% | 29.2% | +37.5 pp | 0.012 |
Our memory metric cannot tell a veridical memory from a counterfeit one — and neither, probably, can yours.
The contribution of this work is not that agents can be fed a false past and will act on it. That is known. It is that the two instruments the field actually uses to check for it would report nothing wrong — and that a protocol which does catch it is cheap, boring, and available on this site.
This site also carries an account of an agent that described the dusk for three days with its camera off, in the author's kitchen, while he was writing a paper about whether machine memory is real. It is the most-read thing here and the most persuasive, and for exactly that reason it needs a label:
It is an uncontrolled field observation, not a fourth demonstration. There is no counterfactual arm, no randomisation, and the confabulation count rests on a keyword filter plus hand-checking. The before/after — roughly forty-five first-person assertions of seeing in 72 hours, then zero — is suggestive, not an estimate. It belongs on this site as the thing that made us look. It does not belong in the evidence column.