Still halfway through reading, but what you've made can unlock a lot of use cases.
> I tried SQLite first, but its extension API is limited and write performance with custom storage was painfully slow
For many use cases, write performance does not matter much. Other than the initial import, in many cases we don't change text that fast. But the simpler logistics of having a sqlite database, with the dual (git+SQL) access to text is huge.
That said, for the specific use case I have in mind, postgres is perfectly fine
SQLite is fine right up until you want concurrent writers. Once you need multiple users, cross-host access, or anything that looks like shared infra instead of a local cache, the file-locking model stops being cute and starts setting the rules for the whole design. For collaborative versioning, Postgres makes more sense.
For a distributed VCS, what would be the need for such things? Even if it were a really big project, how many writes could be going on that this becomes a bottleneck? I don't see it but maybe you have a situation in mind.
In the current environment, even a distributed VCS may have concurrent agents modifying it on different branches.
The problem i faced is mostly importing large repos. But normal use should be fine.
The single-file simplicity of SQLite is a huge win for self-hosted apps. I've been using SQLite in WAL mode for a single-user app and it handles concurrent reads from the API while background workers write without issues. Backup is just cp. For anything that doesn't need multi-user concurrent writes, it's hard to justify the operational overhead of Postgres. ko
Yeah, I get that, and I'm fully on your side. SQLite would have been a nice fit. The only downside is the delta compression problem. Creating an extension for SQLite works, but it's slow. I had two options:
1) Do the delta compression and caching and so on on the pgit side and lose SQL queryability (or I need to do my own), or
2) Use postgres
if you want to use key value store using sqlite then you can try : https://github.com/hash-anu/snkv
in which i am directly accessing b-tree layer and not consuming query layer.
for kv workloads it is much faster compare to sql.
and yeah you will get same benefits of sqlite storage engine.
Also SQLite in WAL/WAL2 mode is definitely not amy slower for writing than Postgres either.
sounds great yes. maybe an SQLite version will come in the future
[dead]