I asked Bubbles — our family AI — why I had rejected one approach to a project months earlier. The answer already existed in my notes. Bubbles answered from the material in front of it instead and missed the decision completely.
This was not an access problem. I had already made years of project notes and operating records searchable. The correct page was in the index. The assistant simply did not reach for it when the question arrived.
An assistant that can search your notes but does not know when to look is still a stranger every morning. Fixing that gap made the system feel less like a chatbot and more like my own memory arriving at the useful moment.
The path there was the setup plus two fixes. Neither fix was the one I expected. Both are worth sharing. They are the difference between “has access to everything” and “uses it”.
Why a brain and not a bigger context window
Two reasonable objections come first.
Why not paste everything into the prompt? Because 596 pages of notes do not fit. And a context full of irrelevant text measurably degrades what the model does with the relevant part. Retrieval is not a workaround for a small context window. Retrieval decides what deserves attention.
Why not a plain vector database? That was my first version: a small custom package with semantic search over the vault. It worked. Then I moved to GBrain, Garry Tan’s open-source personal knowledge brain, self-hosted on my own hardware.
GBrain adds three things on top. First, a self-wiring knowledge graph. Every page write extracts entity references and creates typed links (works_at, founded, advises, attended), with zero LLM calls. Second, typed claims. A fact written in a fenced block becomes a first-class column, so “what was that number in March” is a query, not a rummage. Third, hybrid retrieval with a reranker — the part that decides whether any of this is useful.
The retrieval stack, concretely: embeddings at 1,280 dimensions, plus a keyword index, combined with reciprocal rank fusion. Add multi-query expansion from a small fast model. Add a dedicated reranker model over the top 30 candidates. The provider’s own benchmarks put that at 97.9% recall in the top 5. That is about 31 points better than embeddings alone.
The default install used a cheaper mode without the reranker. I turned the reranker on immediately. It reshuffles about 60% of top-1 results — it is the quality. The full indexing pass cost about two cents. Each query costs three hundredths of a penny. The money does not go here.
Fix one: the corpus poisoned itself
Months in, search quietly turned to garbage — but only for short queries.
I searched for a project decision by its ordinary two-word name. The brain returned journal one-liners. “Beer cover.” “hamburger help me.” Real entries, verbatim, ranked above the page with the decision in its title. I do not know what “hamburger help me” meant. I wrote it.
Everything downstream was healthy. The embeddings were fine — a nearest-neighbor test from a fresh chunk returned perfect neighbors. The keyword index was current. The reranker worked. The sync loop ran. Every part passed, and the whole was broken.
The culprit: about 87 pages of under sixty words each. Old journal fragments and comedy scraps, one line each. Very short text embeds near the centroid of short text in general. So those pages scored 0.86 to 0.97 against almost any short query. That is not a bug in the math. That is what the math does.
The pages flooded the entire top-10. The reranker could sometimes rescue a query. But the query-expansion step generated so many candidates that the true page never survived long enough to reach it.
The most instructive part: the “cheap” mode — raw vectors, no reranker — was the worst of all. That is the exact configuration you would pick to save three hundredths of a penny. It has no defense against this failure.
The fix was corpus surgery, not code. I consolidated 55 journal one-liners into six per-year rollup pages. 18 comedy fragments became one page. I deleted a dozen empty stubs. The content is preserved, and the originals live in git history. Both benchmark queries returned the correct page at #1 again.
The cleanup produced two lessons I would hand to anyone who builds this:
Tiny chunks are as poisonous as tiny pages. I imported my per-project instruction files as a second source — good curated context, clearly useful. But those files chunk into tiny code-fence fragments. That recreates the same centroid pathology one level down. I pulled them out the same day. The content is fine; the shape is wrong, and I do not have a chunking strategy for it yet.
Embeddings drift silently. The audit found 32 chunks with no embedding at all. Nothing flagged them. They were absent from search forever. The sync job now runs an explicit “embed anything stale” pass in every cycle. Also, the query cache serves pre-surgery garbage after a corpus change, so a cache clear is now part of the procedure.
A weekly lint now fails if new tiny pages accumulate. The corpus is an input you must maintain. It is not a pile you dump things in.
Fix two: something must ask
Retrieval was fixed. The assistant still did not use it.
I asked why I had rejected that project approach. It answered from the current conversation and a task title — from everywhere except the brain. The brain had a dedicated page for the decision.
The actual cause is structural, and anyone who wires an MCP-style tool server into an agent will hit it. The brain exposes 102 tools. The agent runtime defers large tool sets behind a search step, instead of a load of every schema into context. That is sensible and necessary at that scale. But the model only discovers the brain’s tools when the message contains a match for them. In practice, that meant: when I typed the brain’s name.
Nobody phrases a real question by naming the memory system first. An ordinary question contains no keyword that surfaces a memory tool. So the model never learned on that turn that the tool existed. It answered from whatever was already in front of it. I had written a directive in the assistant’s always-loaded profile: check the brain first. A directive loses to structure. The model cannot call a tool it was not shown.
The fix is about thirty lines: a plugin that registers a pre_llm_call hook. Before every turn, for any message over sixteen characters, it runs a query against the brain. It injects the top three pages that score above 0.72 into the turn as context.
These details make it safe to leave running:
- Expansion is off (
--no-expand). This is a reflex, not a research task, and it must be cheap. - About 2 seconds is typical, with a hard 8-second timeout.
- Fail-open: if the brain is down, slow, or angry, the turn proceeds with no injected context. It does not error.
- It installs as a user plugin, not a fork of the runtime, so it survives updates.
The model no longer decides to consult its memory. The memory arrives with the question. Your memory works the same way.
I tried the built-in feature for this first. It returned nothing on my corpus. It works from entity aliases, and my pages have no alias metadata. Spend the ten minutes to check the built-in before you write your own. Then do not stop there when the built-in does not fit.
The part that is still fragile
The brain corrupted itself twice inside 24 hours in its first week. A process killed mid-checkpoint left the database unopenable, twice. That story is its own post. Add to that a pinned database dependency: a routine upgrade would silently revert the pin, so every update needed a manual re-pin.
All of that came from one design: an embedded, file-based database with a single-writer lock, held by a long-lived process on a Mac Mini. It does not run that way now. The brain lives in a real Postgres container on the home server. That is a boring answer, and it removes a whole category of failure.
I would tell my past self this. An embedded database is a wonderful choice for something you can afford to rebuild. It is a bad choice for something you treat as memory. The brain was always rebuildable from the vault. That is the only reason those incidents were inconvenient and not catastrophic. The vault is the source of truth; the brain is a derived index. Keep it that way, and you can lose the brain on a Tuesday and shrug.
The moment it became useful
The useful test is still the ordinary question from the opening. I can ask why I rejected an approach months ago, without remembering the name of the memory tool or the title of the note. The right page arrives with the answer.
Other tools now read the same derived index through narrow, read-only access. That is convenient, but it is not the point. The point is that my notes can interrupt me at the moment I am about to repeat an old decision.
Fixing retrieval made the memory accurate. Making it arrive without ceremony made it feel like memory. I still keep the plain-text vault as the source of truth, because Bubbles has already taught me how quickly a brain can disappear.