Agent memory that is just files

The blog this article belongs to has 161 posts and several hundred generated assets. An agent works across all of it, every session, with no vector store and no database.

The memory is the directory tree, and the reason that works is not frugality — it is that almost every question turns out to be exact.

The case this post is about#

One question, asked every session: what produced this image, and from what intent?

$ provenance posts/three-phases/assets/header.jpg
known:  true
engine: stitch
intent: "a mould loft; the chalked line is the authority"
prompt: assets/header.jpg.intent.txt
at:     2026-10-02T11:04:19Z

There is no index behind that. The answer came from a file sitting next to the artifact — header.jpg.gen.yaml — found by appending a suffix to a path the caller already had.

That is the case worth following, because it is the shape almost every real question turned out to have. Not which assets are similar to this one, which would want embeddings, but what is the record for exactly this one, which wants a filename. Across 161 posts and several hundred generated assets, the retrieval that a vector store is for came up rarely enough that the filesystem being the index was not frugality — it was the right data structure.

The symptom#

"Agent memory" usually arrives as a retrieval problem: embed the corpus, search it semantically, feed the results back. That is a real technique for a real class of question, and it quietly assumes the questions are fuzzy.

Here are the ones I actually asked this week:

• which account produced this image, and with what prompt
• did this cover come from the real producer or the fallback
• how many posts are written but not yet illustrated
• what changed in this file since the last run

Not one is semantic. Every one has a single correct answer, findable by name. Running them through an embedding search would return plausible results, which is the worst possible property for a question that has an exact answer.

The failure mode of reaching for the heavier tool here is not that it is slow. It is that approximate retrieval over exact questions returns something confidently wrong rather than nothing, and you cannot tell the difference from the result.

Why the obvious fix failed#

The obvious fix is a database: a table of artifacts, their provenance, their status.

I built one, in the form of a status file recording what state each post was in. Within a day it was wrong, because it was a second copy of information the filesystem already had, and the filesystem kept changing without telling it.

That is the recurring defect with any index over files: it is a cache, and caches need invalidation. Every write to a post is a write that could stale the record, and nothing in the writing path knows the record exists.

The second attempt was to update the record on every write. That is a discipline rather than a mechanism, and it fails the first time something writes through a path nobody remembered to instrument.

The actual mechanism#

Derive the answer from the tree, every time, and let the file be a snapshot rather than a source.

status is DERIVED from disk
the RECORD is not

The status board for these posts is recomputed on every run by walking the directory: does the post file exist, does it have a cover, is there a remote ledger. It cannot be stale, because there is nothing to keep in step — it is a view, not a store.

The written record still exists, and its job is narrower than it looks: it is a snapshot for comparison, so a --check mode can recompute and report what moved. The file answers what changed, not what is true.

Three things the tree gives you free, which an index has to reimplement:

• exact addressing — the path is the key, and a path either exists or does not
• history — every version, every diff, every author, from version control
• co-location — the artifact and its provenance sidecar sit in the same directory, so finding one finds the other with no join

The fix#

Two conventions carry most of the weight.

Per-artifact sidecars, addressed by the artifact's own name. An image at assets/header.png has its record at assets/header.png.gen.yaml. No lookup table, no id, no join — the naming is the relationship, and the question "how was this made" is answered by reading a file whose path you already have.

The whole reader is a path transformation, which is the point — there is nothing to query, nothing to keep in step, and no state that can disagree with the file system:

const sidecarOf = (artifact: string) => `${artifact}.gen.yaml`;

export async function provenance(artifact: string) {
  const path = sidecarOf(artifact);
  const raw = await Deno.readTextFile(path).catch(() => null);
  if (!raw) {
    // Absent is a REAL answer: this artifact was not produced by the pipeline.
    // It is not an error, and it must not be confused with "produced, unknown how".
    return { known: false as const, artifact };
  }
  return { known: true as const, artifact, ...parseYaml(raw) };
}

export async function record(artifact: string, how: Record<string, unknown>) {
  // Written BESIDE the artifact, in the same step that produced it. A record
  // written later is a record that can be missing for the interesting cases.
  await Deno.writeTextFile(sidecarOf(artifact), toYaml({ ...how, at: isoNow() }));
}

Asking how a file was made, months later, with no index and no service running:

$ provenance posts/three-phases/assets/header.jpg
known:  true
engine: stitch
intent: "a mould loft; the chalked line is the authority"
prompt: assets/header.jpg.intent.txt
at:     2026-10-02T11:04:19Z

$ provenance posts/three-phases/assets/legacy-diagram.png
known:  false        # predates the pipeline — a real answer, not a gap

The second result is the one a database makes hard. A missing row reads as a lookup failure; a missing sidecar reads as "this was not produced here", which is usually the truth.

What makes the naming convention load-bearing rather than cosmetic is that it removes the class of bug where the record and the thing drift apart. There is no id to mistype, no foreign key to point at a deleted row, no migration that renames one side. Move the artifact and the record does not follow — which sounds like a weakness and is the correct behaviour, because a record that followed a file to a new location would be asserting provenance it cannot vouch for.

The second convention is that the record is written in the same step that produces the artifact, not in a later pass. A reconciliation pass that walks the output directory and fills in missing records is exactly the thing that produces confident wrong provenance: it knows the file exists and has to guess how it got there, and the cases where it guesses are precisely the cases somebody will later want the truth about.

The honest limitation is that this does not scale to questions that span artifacts. "Which images used this engine version" means walking the tree and reading every sidecar, which is fine at a few hundred files and would not be at a few million. I am not going to pretend that is a general design — it is the right shape at this size, and the threshold where it stops being right is a real one.

What I would keep at any size is the distinction the absent case preserves. A system that cannot distinguish "not produced here" from "produced, record lost" will eventually tell somebody the second when it means the first, and the cost of that is a person spending an afternoon looking for a provenance record that was never supposed to exist.

Query with the tools that are already exact. grep for a field across every sidecar, ls for existence, git log for history. Each returns a definite answer and returns nothing when there is nothing, which is the behaviour you want from a question with one answer.

Concretely, this found something worth finding: grepping one field across every sidecar revealed that 303 of 304 assets had been produced by a fallback path the documentation described as the exception. That is a counting question over a structured field, it took one command, and no embedding would have answered it.

How we proved it#

The direct test is whether the derived view stays correct without maintenance. Across this session the tree changed constantly — posts written, assets generated, six posts held in a draft state while a credential was unavailable — and the board was recomputed each time:

161 post(s) · planned 30 · drafting 0 · ready 19 · medium-draft 29 · medium-live 83

At no point did anyone update that. It is a walk of the directory, so it reported six drafts while six drafts existed and zero the moment they did not.

The comparison mode earns its place separately. Because the written snapshot is only a snapshot, --check can recompute and name exactly which rows moved — which is a question about change, and the only one the file is authoritative for.

The query it cannot answer#

Here is the honest limit, and I hit it in this session rather than imagining it.

Choosing which article to write next required knowing which already-published posts overlapped with a candidate topic. That is a genuinely semantic question — does anything here already make this argument — and the filesystem has no answer to it. Paths do not encode arguments.

What I actually did was grep section headings and read the candidates, which worked because there are 161 posts and I could afford it. At ten times the corpus that approach stops being viable, and the shape of the question does not change — which is exactly where an embedding index earns its keep.

So the boundary is not about scale alone. It is about whether the question names its answer. If you can describe what you are looking for precisely enough to match it by path, field, or string, the tree is better on every axis: exact, current, historied, and with nothing to invalidate. If the question is what here is like this, files cannot help you and it does not matter how few of them there are.

What I would not do#

One tempting middle path I have avoided: keeping an index of some things — a search layer over post bodies only, say — alongside the derived-from-disk view for everything else.

The problem is that it reintroduces the invalidation question for one subset, and subsets are where staleness hides. A record that is wrong about everything gets noticed. A record that is wrong about one category gets trusted for months, because it is right whenever anyone spot-checks it.

If the semantic question becomes frequent enough to need an index, I would rather build it as something explicitly rebuilt from scratch on a schedule — with the rebuild cheap and the staleness window stated — than as something incrementally maintained and quietly drifting.

What this generalizes to#

Before reaching for retrieval infrastructure, write down the questions you actually ask about past work. If they name their answers — this artifact, that field, this path — the filesystem answers them exactly, keeps their history for free, and has nothing to invalidate.

Derive your status views from the tree on every run and keep any written record as a snapshot for comparison rather than a source of truth. Then notice the first genuinely semantic question you ask, because that is the signal the tree has stopped being enough — and it will be a question about similarity, not about scale.