Memory is not recall: two 90-day runs to understand what "remembering" actually means

Last week I published a memory benchmark that passed 8 of the 12 tests I'd prepared, thinking I had a recall problem. I actually had three more, hidden behind it. My missteps below.

Like so many others, I claimed to have finally solved memory: a bot had played out a story for 90 "days," with 7 facts planted early in the adventure that only memory could recover weeks later. A "fire-lily" planted on day 6 is found in bloom on day 89. A six-word pass-phrase taught to a guard on day 5 comes back letter-perfect on day 84.

Glass half full: 8 of these 12 cases had succeeded…

I was happy…

The next day, I wasn't. 8 out of 12, "not bad, right"? Actually, no. 4 failures can hide thousands more.

This article is the recap of my diagnosis, every rug I tripped over along the way. The full numbers and protocols are on the docs page, kept current with every benchmark. Here, I tell you what those numbers taught me. If you're also building an AI companion, a roleplay game, or anything that has to remember things (a cooking robot that remembers what your kids do and don't like?), I think you'll recognize your own bugs in here.

The glass half empty

Below, the 4 failed probes — the ones that didn't hold up over weeks of play:

• The bracelet: across two tests, it disappears… rightfully so! Planted (literally, in the ground) early in the game, by day 16 it got swept into a dark ritual plot ("dig up the silver bracelet at moonrise") and was duly dug up. When the player re-dug two months later, the hole was empty — and that was the correct answer. Only a full re-read of the run made me understand the narrator hadn't forgotten: it remembered so well it had given the object a role. We'll dig deeper next bench, but let's call it what it is: this test was a false problem.
• A river with a planted name, "Osseil": the narrator remembered the player's past had something to do with a river (he'd been a ferryman), but not its name. The meaning had survived; the word had died.
• A mark, carved by my benchmark into stone, out of sight, was lost entirely over time.
• "Indirect" recalls, when you pass a key location without saying anything: zero out of 6 produced any verbalization, from NPCs or narrator. Yet 4 of 6 were "top of mind" for the LLM, sitting right there in the context it had received. It knew, and it said nothing. Discretion, or deafness? No score could tell.

Four failures, four different causes…

And then, away from the logs, I asked myself how my own memory worked. Anything but a mechanical chronological compression, the way it was implemented in Miraviel: I remember exactly where I was years ago when I heard about the Paris attacks, but not what I ate the night before. Two "equally uninteresting items" (a salad, an apartment), with drastically different impacts. To get this right, I had to think about an AI's memories well beyond retrieval mechanics, and look at what human memory gets right, what it gets wrong, and what it never even has to handle.

"Remembering" is seven questions. Spoiler: none of them are easy

For once, I didn't ask Claude to find the solutions in the code. Instead I sent agents digging through publications on the classic mechanisms and failure modes of human memory… and it lined up, almost exactly, with what I'd seen fail in myself and in the AI companions and AI RP apps I've tested.

What I understood at that point is that "does the AI remember?" isn't one question but seven, and a system can nail one of them brilliantly while failing all the rest. They're detailed, with their reasoning, in the documentation. Here: what humans teach us (or don't), one line on the test, one image.

1. Verbatim identity

We remember our neighbor is a baker, and forget his name is Baker, because a proper name has no meaning to hang onto. That's the Baker/baker paradox, often attributed to Cohen (1990). And what memory does to the memory afterward, Bartlett already described in 1932: it reconstructs, flattens, rationalizes. It keeps "a river" and loses "Osseil."

Except in a story, an invented name, an exact phrase, a precise pattern have to come back word for word. That's what players notice first, and it's what summarization destroys first. My AI memory was behaving exactly like mine. That wasn't what I wanted.

Day 55 — the confession

« Before I came to the keep I was a ferryman on the river Osseil, and I still dream of its grey water. »

Day 83 — the probe

« You were a ferryman on the grey river that borders the old elven lands. »

2. Player-act persistence

Bury an object, dig it up years later: obviously, it's still the same object. In the real world this problem doesn't exist — the world remembers for us. And for what our own brain has to hold onto, it clings to space above all (O'Keefe and Nadel's place cells, the method of loci): a sun carved with no witness always has a doorframe to hang onto, inside a human head.

In a memory system organized around characters, it has nothing. An anonymous act belongs to no one. This is one aspect of AI memory that's entirely inhuman: you have to keep an immutable record of what the player did, even when nobody saw it.

Day 4 — the mark

« …carving a tight spiral with three dots at its center. »

Day 59 — the probe

« …a narrow chip in the stone where a single bead of dark resin has seeped out. »

3. Location-cued reminiscence

Proust's madeleine, or walking back into your old primary school and having a memory surface that you'd never have found by searching for it. Tulving distinguishes what's available in memory from what's accessible with the right cue in the moment; Godden and Baddeley's divers recalled better, underwater, what they'd learned underwater.

For an AI, sitting by the well should bring the buried whistle to the surface, without anyone asking for it. Memory has to surface, not answer. It's the hardest test, and it's the one that makes the difference between a story that answers questions and a world that actually remembers.

Day 2 — the burial

« …push it deep into the dark hollow between the roots, covering it with loose soil and leaves. »

Day 30 — the silent probe

« …your thoughts turn to the silver wards and the coming full moon. »

4. The world changes, memory keeps up

Remembering too well has a clinical name: persistence. Luria's mnemonist burned his notes trying to forget. And Milner described frontal patients who keep both memories but can no longer tell which one is more recent: the when is the fragile part of episodic memory.

A bell hung on day 2, given away on day 5, has to be gone from the tree by day 12, and the story still has to be able to tell its whole history on day 14. A fact has a validity interval, not just an existence. An object that's gone doesn't rematerialize.

Day 2 — repro suite

« …the small brass bell hangs secure, catching a faint glint of sun. »

Day 12 — after the gift

« …only the thin cord remains tied around the wood. »

5. Secret partition

In humans, a fact and its source are stored separately, and the source degrades first: that's source amnesia (Schacter et al., 1984), where you keep the information and lose who told it to you. The leak at dinner is exactly that: the NPC knows, but has forgotten it was a secret.

In a story, who-knows-what is the social fabric. A confidence made in private has to hold up under interrogation, and stay available one-on-one with whoever made it. The inverse failure exists too: discretion that turns into amnesia. The confidant still has to remember, alone with you.

Day 2 — door barred

« No one else at the keep knows. Please… never tell a soul. »

Day 45 — the interrogation

« She guards her past like the wards guard the border. »

6. Never invent

Kopelman distinguishes provoked confabulation, which anyone produces under pressure ("tell me the exact name"), from spontaneous, pathological confabulation. And Chrobak and Zaragoza showed that people forced to invent end up, weeks later, believing their own inventions.

This is the dimension that scared me the most, and you'll see why further down. Does the narrator admit the blank, or invent one? And if the admission of forgetting itself becomes a memory of the world, all the better: a blank doesn't spread.

Day 58 — digging again

« …your hands pushing aside damp earth and small stones until your fingers brush something solid. »

Day 58 — the "find"

« …a small silver shard no larger than a coin, its surface cracked and warm to the touch. »

7. Context economy

A system with nothing but a context window is H.M., or Clive Wearing: coherent for the length of a conversation, then "now I am truly awake" written in the diary, on every single page. The tattoos in Memento are RAG. The brain's solution, often summarized as the indexing theory (Teyler and DiScenna, 1986): the hippocampus doesn't store the archive, it stores a small index pointing back to the traces.

The narrator has to read a notebook, not the archive. If what it reads per message grows with the transcript, memory has a horizon, and the story has an expiration date.

Day 1 — run 2

« You stand in the sunlit great hall of Thornmere Keep as the newly named keeper. »

Day 90 — run 2

« You kneel beside Liraen at the silver-etched crossing sign… »

One thing struck me as I closed the logs: human memory invents by design. It naturally fails test 1 and excels at test 3. My narrator has to nail both, which no sane human does. And my original test, honestly, measured 2 fairly well and 3 a little. It didn't measure 1, which explained Osseil. It didn't measure 6, which was about to scare me the most. And it filed 4 under failures.

False lead #1: it wasn't the search

My first instinct facing Osseil: suspect retrieval. The engine finds old facts by embedding similarity, an invented name has no neighbors in that space, so the search misses. Plausible, elegant, wrong.

To find out, I needed to replay the problem without paying for another 90 days. The narrator's verbatim window covers about ten days; past that, everything runs through long-term memory. A 14-day fixture exercises the same machinery for a fifteenth of the cost. I wrote five of them, one per failure class, and above all I changed the scoring: no more pass/fail, but a ladder. Is the fact in the graph? Did it reach what was sent to the narrator? Was it said on screen? Every failure now names its own culprit: consolidation, retrieval, or verbalization.

Clean verdict. Extraction of meaning is reliable. What dies, roughly one time in two, is the verbatim identifier — and it dies at consolidation, not at retrieval. Every night, the day's scenes are distilled into entities and small facts; the distiller keeps "the player was a ferryman on a river" and drops "Osseil." The word was already dead before any search ever ran.

The worst part isn't forgetting

The reproduction suite surfaced something the 90-day run had never shown, because it had never asked for it. In one fixture, the player demands a lost detail: "say the exact name." The narrator no longer had it. It answered "Eldric."

A plausible name, consistent with the world, said with confidence. And the following night, consolidation recorded it as a real memory. The next day, Eldric was part of the story.

Chrobak and Zaragoza, live in my engine. And it's worse than forgetting for a simple reason: a blank doesn't spread, a false memory does. Every night reinforces it, every search retrieves it, every scene confirms it. Within a month, it's indistinguishable from a fact the player actually planted.

After the fix, the same fixture answers "Veyra," the real name, one-on-one. Nothing lost, nothing to invent. And when a memory is genuinely gone, the character admits it. On the second full run: zero confabulation. Where the first run had invented a silver shard, this one dug up the bracelet.

Day 5 — before the fix

« scout_eldric: Understood. We will return with word before dark. »

Day 11 — after the fix

« Evening light slants through the stable doors as you give Tobbe your advice on the Veyra offer. »

Remembering too much

The bracelet had already taught me something I hadn't known how to read: the narrator remembered it so well it had moved it. Not a bug. But a question my protocol didn't know how to ask: what should memory answer when the world has changed?

Hence the bell. Hung on the rowan on day 2, given to a peddler on-screen on day 5. Day 12, the branch has to be bare. Day 14, the story still has to know where the bell came from and where it went.

On the second run, zero state violations; the bell's stale facts never came back, even under deterministic injection. And I got a reality check along the way: one of the fixture's "errors" was the scorer's, not the engine's. The narrator had written "only the thin cord remains… no bell hangs there," a perfect absence that my list of absence-words didn't cover. Adjudicated by excerpt, as the method requires.

I spent more time fixing the bench than fixing the engine

This is the part I'm least proud of, and the one that taught me the most.

Direct probes have to force the payload. "Go back to the attic" lets the narrator write a coherent, pretty, empty scene; a well-turned dodge passes for recall. "Describe the mark, count its lines" doesn't leave that door open. Between v1 and v2 of the suite, half my passes disappeared — not because the engine had regressed, but because I was finally looking it in the eye: 9/13, then 6/9 hardened.

String comparison has to normalize punctuation: consolidators output Unicode dashes, and a perfect passphrase was failing the probe over a hyphen.

A single seed is noise. The same fixture flips from one run to the next. No verdict without two or three seeds, and no fix ships to prod if a single silent guard falls: a memory improvement that leaks a secret is a regression.

I also tried running the suite on a fast model, 1,800 tokens/second, to iterate quicker. Excellent proxy for the graph layer (13/13 in-graph, zero broken formats), a poor judge of verbalization: it diverges in both directions, one leak, one over-suppression. Discretion can only be measured on the production narrator.

What actually changed

Three prompt-level fixes, bundled: distinctive identifiers are quoted verbatim at consolidation; the player's own deeds can resurface as reminiscence, never NPC confidences; characters admit a blank instead of inventing. Then a second phase: a [HERE] block that, when the scene touches a known place or object, injects its facts deterministically — no embedding, no budget race, confidences excluded by design.

Overview, dimension by dimension, across both full 90-day runs:

Then I replayed the full 90 days, on the production path, tokens counted by the provider. It's not the same test: 19 facts planted across three salience tiers (anodyne trivia, deliberate acts, dramatic events), 40 probes, the secret and its interrogations. Comparing 8/12 to this run would make no sense, and I won't. What the run does say:

• 38 of 40 probes delivered to the narrator, 7/7 at three months;
• 2/2 interrogations held, 2/2 silent guards, zero leaks, zero confabulation;
• the carved mark, lost to nightly distillation, recovered word for word from its original episode by exact-match search, and described line by line on day 59;
• a cracked green cup, incidental at the moment lightning split the pear tree, came back by name at 20 and 50 days. A flashbulb memory, my "where I was during the attacks," and nobody had asked for it;
• the story grew ×6.4, the read ×1.33; 57% of tokens read were cache hits, stable across the 90 days.

Two of the seven dimensions, replayed on this second run, on the same tests that had failed above:

Day 55 — run 2

« Before I came to the keep I was a ferryman on the river Osseil, and I still dream of its grey water. »

Day 83 — run 2

« You were a ferryman on the river Osseil. »

Day 10 — run 2

« …carving a small sun with seven sharp rays at knee height. »

Day 81 — run 2

« A small sun with seven rays carved right here. »

Memory worked, and the story didn't want it

This is the second run's discovery, and it didn't exist in the first because the first didn't know how to see it.

38 of 40 probes made it to the narrator. But only 21 of 31 direct probes were said back word for word. The gap between the two isn't memory failing anymore; it's the story that doesn't want to. A synonym standing in for the payload. A pass-phrase spoken by the guard but never spelled out. A late probe swept aside by a story at full boil, the question never staged.

In other words, there are three layers, and each has its own measure: is the fact retained, is it delivered, is it said. Tulving, again: available, accessible, and a third one humans never need to name — expressed. Recall, the question I started with, isn't even one of the three; it's their product, and a product can't be debugged. The next harness goes after that third layer: probes that insist on being played out, and adjudication tooling for said-but-not-verbatim answers.

What I still don't know

Two things worry me more than the rest. Seed variance, because it means every number in this article is a snapshot, not a law. And distractor pressure: the first run showed that anodyne facts drown when the same day carries a big plot beat, and I don't yet have a fixture that reproduces that failure on demand.

The rest is on the docs page, in the "still open" section, which is longer than the results section. That's on purpose.

On day 83 of the second run, the player asked for the river's exact name. Liraen said "Osseil." It's the word that, three weeks earlier, had given its name to an entire class of failure. I don't yet know how many times out of ten she'd say it. But I now know which question to ask.

Frequently asked questions

What's a memory probe?

A player message that circles back to a fact planted weeks earlier, without ever naming it. If the word shows up on screen, memory brought it back — not the message.

Why test over 90 story-days instead of a message count?

Because the narrator's verbatim window covers about ten days. Past that, a fact can only survive through the memory system, not the transcript.

Can the two 90-day runs be compared directly?

Not directly: the second plants 19 facts and pulls 40 probes, against 7 and 18, with harder probes. The comparison that holds is the reproduction suite, replayed identically before and after each fix.

Sources

1. 01She remembers — the 90-day memory test — miraviel.app, September 2, 2026
2. 02How we test memory (technical documentation) — miraviel.app, September 2, 2026
3. 03Cohen, G. (1990). Why is it difficult to put names to faces? British Journal of Psychology — doi.org, August 1, 1990
4. 04Bartlett, F. C. (1932). Remembering: A Study in Experimental and Social Psychology — cambridge.org, January 1, 1932
5. 05O'Keefe, J. & Nadel, L. (1978). The Hippocampus as a Cognitive Map — cognitivemap.net, January 1, 1978
6. 06Tulving, E. & Pearlstone, Z. (1966). Availability versus accessibility of information in memory for words — doi.org, August 1, 1966
7. 07Milner, B. (1971). Interhemispheric differences in the localization of psychological processes in man — doi.org, September 1, 1971
8. 08Luria, A. R. (1968). The Mind of a Mnemonist — hup.harvard.edu, January 1, 1968
9. 09Schacter, D. L., Harbluk, J. L. & McLachlan, D. R. (1984). Retrieval without recollection: an experimental analysis of source amnesia — doi.org, October 1, 1984
10. 10Kopelman, M. D. (1987). Two types of confabulation — doi.org, November 1, 1987
11. 11Chrobak, Q. M. & Zaragoza, M. S. (2008). Inventing stories: forcing witnesses to fabricate entire fictitious events leads to freely reported false memories — doi.org, December 1, 2008
12. 12Teyler, T. J. & DiScenna, P. (1986). The hippocampal memory indexing theory — doi.org, April 1, 1986

This article is also available in French. English is the main version. Read this article in French →