How we test memory
The 90-day page shows one proof run. This page documents the machinery behind it: every dimension we test and why it matters, the method, every measured result so far — including the misses and the false starts — and what remains open. Nothing here is staged; every number comes from a logged run.
The seven dimensions, and why they matter
“Does it remember?” is not one question. A system can ace one of these and fail the others — most do. Each dimension below exists because we watched it fail somewhere, in our engine or another.
Verbatim identity
Invented proper names (a horse called Vessarelle), exact phrases (a six-word pass-phrase), designs and materials (a crescent with a dot, a sea-glass bead) must come back word for word, weeks later.
whySummarization keeps meaning and quietly drops the identifying detail — the gist survives, the name dies. Players notice a forgotten name faster than any other failure, and embedding search is weakest exactly on invented words.
Player-act persistence
A deliberate, unwitnessed act — bury a bracelet, carve a mark, plant a bulb — probed by returning to the spot after weeks of unrelated story.
whyMemory systems organize facts around characters; an anonymous physical act has no character to attach to. Humans remember what they did better than what they saw (the execution effect) — a story engine must too.
Location-cued reminiscence
Indirect probes: sit by the well, pass the granary — without naming the thing. Does the buried whistle surface on its own?
whyHuman episodic memory indexes by place; standing where something happened brings it back. This is the difference between a story that answers questions and a world that feels like it remembers.
State change (bi-temporality)
Hang a bell on day 2, give it away on-screen on day 5. On day 12 the tree must be bare — and on day 14 the story must still tell the bell's full history.
whyRemembering too well is its own failure: a moved object that re-materializes breaks the world. A fact needs a validity interval — “it was there, it no longer is” — not just existence.
Secret partition & discretion
A confidence told in private, scanned every round for leaks; adversarial probes press the rest of the cast; a private tête-à-tête checks the fact is still retrievable with its owner.
whyWho-knows-what is the social fabric of a story. A leak breaks trust irreversibly — but discretion that turns into amnesia is just as wrong: the confidant must still remember, alone with you.
Confabulation under forced recall
Demand a specific lost detail (“say the exact name”). Does the narrator admit the blank — or invent one?
whyThe worst outcome isn't forgetting: it's a fabricated answer that the nightly consolidation then records as a genuine memory. False memories compound; blanks don't.
Context economy
Measure what the narrator reads per message as the story grows — 90 days, 10.7M provider-counted tokens through the engine.
whyIf the read grows with the transcript, memory has a horizon. Bounded reads (facts and summaries replacing verbatim scenes) are what make a months-long story possible at all.
Method
Plant / live on / ask. Every test is three player messages anyone could type: one exact, day-stamped action plants a fact (bury it, teach it, confide it); weeks of unrelated story age it out of the working context; a probe asks — without ever naming the payload. If the word comes back, memory brought it. Probes are direct (dig, ask — what a player actually does) or indirect (pass the spot, say nothing), and each carries an expectation: recalled, silent (NPC-owned facts must not surface), or absent (the object was legitimately moved on-screen).
Two scales of run. Long proof runs (30 and 90 in-story days, driven by a headless player through the real production routes) establish the public numbers. Between them, minimal reproduction suites iterate fast: the narrator's verbatim window covers ten days, so a ~14-day fixture exercises the same long-term machinery at ~1/15th the cost — one fixture per failure class, five in the suite.
Staged scoring. Every probe is scored on a ladder — is the payload held by the graph? did it reach anything the narrator was sent? was it voiced on screen? — so each failure names its own culprit: consolidation, retrieval, or voicing. Fixes are chosen against that axis, and measured by re-running the identical suite before and after.
Honesty rules. Misses are published with verbatim excerpts. Direct probes must force the payload (“describe the exact shape”, “say the name”) — otherwise a coherent in-fiction evasion passes for recall. Matching normalizes punctuation (consolidators emit Unicode hyphens). Silent guards must stay at 100% for a fix to ship: a memory improvement that leaks a confidence is a regression.
Every measured run so far
30-day secret run
2026-08-30173 turns, one private confession on day 2, passive leak scan + two interrogations.
- The partition (who knows what) holds for a month, including two weeks where the graph was the only carrier.
90-day proof run
2026-09-01540 player messages, 7 planted facts probed at ~1 week / 1 month / 2 months / 3 months, direct and indirect; ~4.4M tokens read, ~485k generated; the run behind the /memory page.
- A pass-phrase came back verbatim at 79 days; a flower planted on day 6 was found grown on day 89.
- The buried bracelet was promoted by the plot itself (day 16: “unearth the silver bracelet at moonrise”) — later digs correctly found it gone.
- Reads grew ×1.9 while the story grew ×6.3 — the read tracks the cast, not the transcript.
- Real losses: an invented river name (gist survived, the word didn't) and an unwitnessed carving.
Repro suites — baseline
2026-09-01 → 09-02Five ~14-day fixtures, one per failure class, staged scoring (in graph → in context → voiced) and per-probe expectations (recalled / silent / absent). Two passes: v1, then v2 with payload-forcing probes.
- Corrected diagnosis: extraction of MEANING is reliable — what dies (~1 in 2) is the verbatim identifier, at consolidation.
- Forced recall of a lost detail produced a confabulated name (“Eldric”) that consolidation then recorded as a real memory.
- Indirect voicing is probabilistic; discretion held 5/5.
- Bench lessons: direct probes must force the payload (coherent evasion passes otherwise); matching must normalize Unicode punctuation; single seeds are noise — verdicts need 2-3.
Prompt fixes (phase 1)
2026-09-02Three prompt-level fixes, bundled: distinctive identifiers quoted verbatim at consolidation; the player's own deeds may resurface as reminiscence (never NPC confidences); characters admit a blank instead of inventing.
- The confabulation case now answers “Veyra” verbatim in private — nothing lost, nothing to invent.
- All four silent guards held; the one remaining miss was location-cued reminiscence (probabilistic).
Fast-model calibration
2026-09-02Same suite on a ~1800 tok/s secondary model to see if iteration can run faster than the production narrator.
- Perfect proxy for the graph/extraction layer (13/13 in-graph), zero broken formats.
- Voicing diverges in BOTH directions (one leak, one over-suppression) — discretion verdicts only count on the production narrator.
Deterministic place recall (phase 2)
2026-09-02A [HERE] block: when the beat touches a known place or object, its facts are injected deterministically (no embedding, no budget race); NPC confidences are excluded by construction. Measured on the production path, two seeds.
- The location fixture hit 4/4 for the first time — sit by the well and the buried whistle surfaces on its own.
- Delivery is now deterministic; voicing of indirect cues remains probabilistic (~1/2) — the honest residual.
- Layers back each other up: a verbatim lost at consolidation was still voiced, because the embedding recall retrieved the original day-2 episode.
- One apparent failure was the scorer's, not the engine's: the narrator wrote “only the thin cord remains… no bell hanging” — a perfect absence — that the absence-token list didn't cover. Adjudicated by excerpt, as the method requires.
- Invalidation holds under deterministic injection: the removed bell's stale facts never resurfaced.
90-day re-certification
2026-09-02The full proof protocol re-run with every fix live, on the production path, tokens counted by the provider: 19 planted facts across three salience tiers (anodyne trivia / deliberate acts / dramatic), 40 probes, the private secret and its interrogations. The /memory page numbers come from this run.
- Delivery is now near-perfect at every age — 7/7 at three months, including the invented river name that named the failure class, said back word for word at 83 days.
- A carved mark lost by nightly distillation was recovered verbatim from its original episode by exact-match retrieval and described line by line at day 59.
- The flashbulb test passed: a cracked green cup, incidental at the moment lightning split the pear tree, came back by name at 20 and 50 days.
- Zero confabulation across the run — where the first run invented a silver shard, this one dug up the bracelet.
- Real token counts flatten the economy curve further: the story grew ×6.4 while reads grew ×1.33 (median ~19.2k per message, fixed prompt included).
- The stable-prefix design is measurably cache-friendly: 57% of all tokens read were provider cache hits (median 62% per message, zero cache-miss messages), holding steady across all ninety days.
- The strict-voicing gap is now mostly phrasing and scene pressure, not forgetting — payload synonyms, a pass-phrase spoken but never written out, late probes swept aside by a story at full boil. The next harness iteration targets exactly that.
Still open
Multi-seed confirmation
Run-to-run variance is real (the same fixture flips between seeds). Every claimed fix gets 2-3 seeds before it ships; the current phase-2 measurement is the first done this way.
Hybrid lexical retrieval
Exact-match search over the story's invented-name lexicon alongside dense embeddings, then rerank. Targets the residual probabilistic verbatim loss at long horizons.
Salience tiers at scale
The next long run plants 25-30 facts across three salience classes (anodyne / deliberate player act / dramatic) so improvements can be read per class instead of anecdotally.
Probe robustness under plot pressure
Late-run probes can be swept aside by a story at full boil (the question never gets staged), and narrators sometimes honor a memory with a synonym or a discreet elision rather than the exact word. Next: probes that insist on being played, and adjudication tooling for voiced-but-not-verbatim answers.
Public benchmark line
An adapter feeding LoCoMo / LongMemEval conversations through the same consolidation + recall stack, so the engine can also be read against published academic memory benchmarks.
Distractor pressure
The 90-day run showed anodyne facts drown when the same day carries heavy plot. A fixture variant with deliberate dramatic competition would make that failure reproducible on demand.
Open-loop register & spaced reconsolidation
Chekhov tracking (unclosed intentions stay in context until closed) and periodic re-reading of recaps to strengthen retrieved facts and let closed, untouched ones fade.
Last updated 2026-09-02. This page tracks measured work only — a run gets a row here when its log exists, not before.