I built this with simple text-based memory archive too. What you said is simply adding git to the equation. I tried many approaches and, to my surprise, Markdown + Git + plain-old UNIX tooling is powerful.

I've noticed that anchoring the tool on well thought out standards correlates with good performance.

Concretely: using Markdown, JSON, RFC 822 MESSAGE ID for identifying emails, or using self-contained binaries (or simply executable files with UNIX shebangs) are all instances of where I've converged after many attempts at using more complex techniques. Examples of those techniques are PostgreSQL, XML, trying to recreate what's essentially Git (for the time component), and even embeddings in some cases.

I think this is an instance of worse-is-better.