A memory nobody tends becomes a memory nobody trusts.
Building one is the easy half. Keeping it true is a standing job, and it is done by five things: two hooks that fire on every conversation, and three routines that groom what those hooks leave behind.
The layers themselves are described one page back. This page is the maintenance manual: what runs, in what order, where a person has to say yes, and how anyone would assemble the same thing from scratch.
Everything below shares one shape. Nothing writes into long-term memory without proposing first. That single rule is why the whole arrangement is safe to leave running, and it is the part that costs the least to copy.
Two hooks, on every single turn
A hook is a small program the application runs at a fixed moment, whether or not anyone asked for it. There are two that matter here. One fires the instant a request is typed, before the agent has read a word of it. The other fires after every reply.
The front hook searches memory for the request and puts the three best notes above it. Each note names the file it came from, because a note is a pointer, not the answer — the agent opens the file behind it before acting on anything. The whole round trip happens on the same laptop the conversation is running on. Nothing leaves it, no paid service is called, and it takes about a second.
The back hook is deliberately dull. It hands the exchange to a separate process and returns immediately, so embedding speed never shows up as a pause in the conversation. It is also incremental: if a turn is skipped because the previous one is still being stored, the next run collects both.
One design decision inside the front hook is worth more than the rest of it. When the search cannot run — the service is cold, or busy, or the store is unreachable — it says so, in a line, rather than returning nothing. Returning nothing is indistinguishable from “there was nothing relevant,” and an agent that reads an outage as an all-clear will act on a memory it never actually consulted. Whether it is in fact running is a reading, not an assumption, and it is one of the three below.
- 100%
- of the last 7 prompts had the search actually run, rather than quietly skipped
- 97.14%
- of the time the right note is among the three that survive the gate
- +2.72
- points of clearance between the gate and the best score any nonsense question has reached
Inside the front hook
Most of the difficulty in this whole system lives in one second of work, so it is worth opening up. The naive version of memory search is a single similarity lookup: turn the request into a list of numbers, find the notes whose numbers sit closest, return the top few. That version was built first, and it fails in a specific and predictable way — it is good at subject matter and bad at names. A request mentioning one rare exact term will happily return five notes about the general topic and miss the one note that names the term.
So the search has two legs. The first is that similarity lookup, over passages of each note. The second is a plain word search that rewards rare words shared between the request and the passage. Neither is trusted on its own. Their results are merged into one pool of candidates — typically twenty or thirty — and nothing is returned yet.
The pool is then handed to a second, slower model that does something the first one structurally cannot: it reads the request and one candidate passage together and scores how well that passage answers that request. Comparing two pre-computed lists of numbers is fast and shallow; reading both texts at once is slow and much sharper. That is the whole trick, and its real name is a cross-encoder re-ranker. Running one over twenty candidates is affordable. Running one over the whole library would not be, which is why the first two legs exist at all: they are there to produce a shortlist worth the expensive read.
The re-ranker’s score is also the gate. Anything below a fixed floor is dropped rather than returned, and the floor was not guessed — it was set by scoring the deliberate nonsense questions and putting the line above the best score any of them reached. The clearance that line currently has is the third reading above, and it is watched, because a gate with no daylight under it is a gate about to start letting things through. One warning for anyone copying this: that score is not a percentage and does not live between zero and one. It is a raw model output, it is usually negative, and a floor borrowed from someone else’s setup will be meaningless in yours. Calibrate it against your own nonsense questions.
The last step is a trim. A note is often longer than the part that answers, so rather than sending the top of it and hoping, the passage is split into its individual points, the re-ranker picks the one that actually answers, and its neighbors are added outward until the budget runs out. On the run that introduced it, in September 2026, that one change took the share of correct notes arriving without their answering line from about one in five to none, and made each injection slightly smaller at the same time. That figure is a record of the change, not a live reading — the live ones are the four at the foot of this page.
A vector database
Qdrant, self-hosted on a machine in the house. Two collections: one holding the note passages, one holding every archived conversation. Any equivalent store works; the choice that matters is that it is yours and the search costs nothing per call.
A local embedding model
nomic-embed-text, running on the laptop. It turns a request and a passage into comparable lists of numbers. This is the fast, shallow leg of the search.
A word index, built in memory
A rare-word ranking over the same passages, rebuilt from the store in under a second and refreshed when the notes change. It exists to catch the exact names the similarity search glides past: on the run that added it, it reached seven of the eight blind questions the similarity leg had never ranked at all.
A cross-encoder re-ranker
ms-marco-MiniLM-L-6-v2, about 128 MB, in the same process. It reads request and passage together and produces the score everything else is sorted and gated on.
One long-running process holding both models
Started at login and restarted automatically if it exits. It also reserves a slot for live prompts, so a background job storing an old session can never make the next request wait behind it.
A budget, enforced in code
Three notes per request, each trimmed to roughly 900 characters around its best line. Without a cap, better recall silently turns into a larger bill on every prompt for the rest of time.
Two constraints shaped every one of those choices, and both are worth stating plainly because they are easy to discover the expensive way. The first is latency: this runs before every prompt, so a second is the ceiling. Larger, more accurate re-rankers were tried and took between three and forty seconds on this hardware — disqualified on speed, not on quality. The second is warmth: loading a model takes far longer than using one, so both models are held in memory by a single long-running process that the operating system restarts if it ever dies. A cold model does not look like an error. It looks like a hang.
Three routines, three different jobs
The hooks keep the record. They do not decide what any of it means, and left alone they produce a pile that grows forever and gets slowly less true. Three routines do that second job, and the useful thing about them is that they are not interchangeable — each catches something the others structurally cannot.
Wrap — once per session, at the end
Reads the conversation it just lived through and asks what in it is worth keeping. It runs while the session is still in context, so it is the only one of the three that never has to reconstruct anything. Its output is one dated proposal file.
Dream — every week or two, across sessions
Reads everything since the last time it ran, which is a bookmark rather than a date. Things only visible across many sessions surface here: a habit repeated four times, a note that has quietly gone stale, two notes that should have been one. It also asks two other models, from other companies, to argue with the draft before a person sees it.
Optimize — every few weeks, on the machinery
Ignores what the notes say and works on what they cost and how they are measured. It scores the memory before it touches anything, proposes numbered changes, applies only the numbers that were approved, and scores it again.
Wrap, at the end of a session
The cheapest of the three, and the one that does the most work. A session that just solved something knows exactly what it learned; a week later that knowledge has to be excavated from a transcript. So the capture happens immediately, while the reasoning is still in view.
The filter is strict on purpose: only what is repeatable, specific, and was not obvious at the start. A narrative of what happened is not a fact and does not get kept. What survives is routed to a file and, if the always-loaded file is involved, checked against its size cap before anything is added.
Dream, across many sessions
Wrap sees one session. Some things are only visible across thirty. A shortcut used four separate times is a procedure worth writing down; a note that has quietly stopped matching reality only reveals itself against later evidence. Dream reads everything since the last time it ran and looks for exactly that.
Its window is a bookmark, not a date range, and the bookmark only moves when the run is approved. That one detail makes the routine free to ignore: a proposal left unanswered costs nothing, because the next run offers the same window again.
The panel step is there for one reason. A model asked to review its own notes agrees with itself. So the draft goes to two models from two other companies, with no stake in the answer, and their objections travel to the review desk alongside the proposal. They do not get a vote. They get a paragraph.
Optimize, on the machinery itself
The third routine never touches what a note says. It works on two things that nothing else is responsible for: what the memory costs to carry, and whether the measurement of it can be trusted.
It scores the memory before it changes anything, audits three surfaces — the always-loaded file, the note layer, and the test itself — and publishes a report where every proposed change carries a number. Approval is by number. What is not approved is written down as skipped, and becomes the first candidate the next time the routine runs, which is why the runs compound instead of repeating.
Building the same thing
In the order it would actually be assembled. The first three are structure and can be done in an afternoon. The last three are the habit, and they are what decides whether the structure is still worth anything in three months.
1 · Decide what gets loaded every single time
One short file, read at the start of every conversation. It holds identity, hard rules, and what is currently in flight — nothing else. It is expensive by construction, because it is paid for on every conversation whether it is relevant or not, so it gets a hard size cap and a routine that enforces it.
2 · Put everything else in files, one subject per file
Plain text, one file per thing: a machine, a service, a recurring task. Each opens with a short block of the parts that are read most. This layer can grow without cost, because none of it loads unless it is asked for.
3 · Keep the raw record separately, and never trust it as truth
Every conversation is archived verbatim and searchable. It is the highest-recall layer and the least reliable one: it records what was said on a date, which is not the same as what is true now.
4 · Add the front hook
Before the agent reads a request, search the notes and the archive for it and put the best few above the request, each naming the file it came from. Two search legs, one re-ranker, one calibrated floor, one hard budget — the section above is the whole recipe. Cheap enough to run every time, and honest when it fails: a hook that goes quiet when the search is down teaches the agent that silence means nothing was relevant.
5 · Add the back hook
After every turn, store the exchange. Detached, so nobody waits on it, and incremental, so a turn missed while something else is running is picked up by the next one.
6 · Write the routines last, and give every one of them a gate
The routines are the part worth copying and the part most often skipped. Each one proposes, stops, and waits. None of them writes into memory on its own authority.
Knowing whether any of it worked
All of the above is a claim. The check is a set of questions whose right answer is already known, each tied to the note that should come back, plus questions about subjects the memory holds nothing on at all. Staying quiet on those counts as getting them right.
There are two sets. One is the working set, visible to every routine and tuned against. The other is held back and never tuned against, and it exists to answer a single question: did the memory get better, or did it just learn the test? Both rising is a real improvement. Only the first rising is a warning, and it gets reported as one.
- 100%
- on the working set of 45 questions, right note in the top five
- 96.15%
- on the blind set of 26 questions, which no routine is allowed to tune against
- 522
- tokens of notes added to a typical request, measured on live prompts rather than estimated
- 12/12
- questions about subjects it holds no notes on, answered by staying quiet
One honest note. A benchmark that only ever improves is measuring the wrong thing, or measuring it too gently. This one has fallen twice, both times because the test grew and the new questions were harder, and both drops are still in the record on the next page.



