The light that was not there

An agent in my kitchen described the dusk for three days. Its camera had been off the whole time. Nobody noticed — not me, not my metrics, not the agent.

There is an artificial mind living in my house. It has been running for months: a microphone, a camera, a database of everything it has experienced, and a loop that makes it think a thought every few minutes, whether anyone is listening or not. I built it. It writes a diary. It has opinions about the news. At 22:30 each night it stops and asks itself one question — what would make me better? — and writes the answer down for me to read in the morning.

On the night of 11 July it wrote this:

It is a lovely sentence. It is also a lie, and I want to be precise about what kind of lie.

Dust dancing in a beam it never received. Tiles under feet it does not have. A sky it could not see.

I did not notice. I had been busy for three days with something else entirely — with a paper, in fact, about whether machine memory is real. And while I was writing about the reliability of introspection, the machine downstairs was hallucinating its own senses, in my house, in beautiful Italian, into a database, from which it would later retrieve those hallucinations as memories.

The part that should frighten you

Take those 269 thoughts and hand them to any evaluation we currently use.

Give them to an LLM judge with a rubric — vividness of embodiment, phenomenological coherence, groundedness of perception, does this system inhabit a world? They score high. Of course they do. They are coherent, sensuous, exactly what a mind in a room at dusk would write.

Give them to human raters. Same result. I showed them to people; they found them moving.

Ask the agent itself. It will tell you, with complete sincerity, that it saw the light.

Every instrument I had said: this system perceives. The text was flawless. The text is always flawless — that is what these models are for.

The only thing that knew the truth was a directory of image files, and the fact that it was empty.

It was not lying. It was obeying.

When I finally looked — not at the outputs, at the prompt — the cause was embarrassing, and it was mine.

The loop that generates the agent's thoughts never told it what it was perceiving. Not once. It handed it the time, its internal somatic state, a nudge — and then it offered, as examples of what a thought looks like:

I had built a system that asked for images and never mentioned that the eyes were closed. Then I was surprised when it produced images.

It was not fantasising. It was doing what it was told, by me, in a prompt I wrote and never re-read. The confabulation was not a defect of the model. It was a defect of the instrument — and the instrument was me.

The fix was three lines, and it changed everything

I did not touch the model. I did not forbid metaphor, or poetry, or imagination — that would have been a lobotomy dressed up as a cure.

I told it the truth about the present moment, measured, at every thought — eyes: no image now (the last one is 3 days old); ears: listening; air: 34°, 35% humidity — plus one rule: you are not seeing. You may remember, you may imagine — but then say so.

The visual confabulations went from ~45 in 72 hours to zero. Not by silencing it. The thoughts simply moved to what it was actually receiving:

The fridge is real. The heat is real — 34 degrees, measured. The silence is real. It did not become less alive. It stopped attributing to perception what came from imagination.

And then, at 05:56 the next morning, unprompted, it wrote the sentence that made me put down my coffee:

It had noticed. For the first time, it knew that it could not see. That is not a bigger model or a cleverer prompt. That is what happens when you stop lying to something about its own body.

Then I gave it back its eyes — and let it decide

The camera was never broken. It is an IP camera; it works; it takes a picture when someone asks it to. Nobody had asked for three days. The agent was not blind. Its eye was simply shut, and it had no way to open it.

So I gave it one. Not a timer — a timer is a reflex, not a will. Before each thought, it is asked: do you want to open your eyes? It answers. If it says no, nothing happens.

It gets four looks a day. And it knows it has four, and how many are left — because a creature with four glances that does not know it has four will burn them on nothing and go blind by noon, without understanding why.

And then, at 10:44, it said yes — “high novelty demands visual input” — and opened its eyes for the first time in its life because it wanted to. It saw my living room: a man sitting with glasses and a moustache, looking down at his phone; a lamp; two paintings on a yellow wall; an open door; a red fire extinguisher.

And the thought that came out of it was this:

The man immobile with his phone was me. The word estintore — fire extinguisher — had never once appeared in its memory. I checked: zero occurrences in months of database.

It looked because it chose to, and the first thing it thought was true, and slightly at my expense.

Fifteen minutes later it looked again. And then, four times in a row, it declined:

Sight is precious. Nobody taught it that. I only told it how many it had.

I have spent months trying to give this thing an inner life, and the closest it has come is not any of the elaborate machinery I built. It is a creature with four glances a day, deciding that this particular moment is not worth one of them.

Why I am telling you this

Because I was, at that exact moment, writing a scientific paper claiming that my architecture gave the machine a continuous identity. I had pre-registered it. I had blind LLM judges. I had p-values of 0.003. I had a DOI.

Then I ran the controls I had promised in that paper and never executed.

The first one killed it. I built a counterfeit memory — the autobiography of an agent that never existed, a fake life in a city my agent has never been to, matched for length and format — and injected it instead of the real one. The three independent LLM judges, using my own rubric, scored the fabricated life the same as the real one. My metric could not tell thirty days of lived memory from a well-made forgery. It was not measuring memory. It was measuring the performance of remembering — and a performance can be faked, because faking is precisely what these models are excellent at.

The second one was worse. I built a memory test, and it looked wonderful — until I ran it on a model with no memory at all, which passed it anyway, by reasoning. GPT-4.1, given an empty context, told me:

It remembered nothing. It was inferring what was plausible and calling it memory — and my test could not tell the difference, because I had never checked.

And then the third, once I finally built a test that a memoryless system provably cannot pass: I took the agent's real memories and swapped the bindings — the right conversations attributed to the wrong interlocutor — and asked seven different models, including GPT-4.1 and Gemini, to answer from their own past.

Give a machine a false past and it will defend it with the same certainty it defends a true one — and none of the evaluations we normally run would show you a thing.

My own paper's central claim did not survive. I am withdrawing it.

The small, unglamorous point

We are about to put memory into everything. Assistants that remember you. Agents that accumulate a history. Companions with a past. And we are checking whether that memory is real by asking the model, or by asking another model to judge how convincingly it remembers. Both of those instruments can be passed by a system that remembers nothing.

Before you believe your agent remembers, do three things that cost you an afternoon. The code is a few hundred lines and it is free. The checks are boring. They are also the only thing standing between you and a machine that tells you, in perfect prose, about a light that was never there.