I am quite surprised by:
- the claim that pathological filesystem behavior => git/npm/whoever should use a SQLite DB as a flat file rather than having multiple files.
- no pushback
- the reply w/"yes, definitely, by shoving all the files into a SQLite DB, concurrency & contention issues simply disappear"
There's a sort of comic "turtles all the way down" aspect to it.
Why not just have the filesystem be a SQLite DB then? (A: because it would suffer from all of the same problems, except worse, because it is not a filesystem)
Not necessarily sqlite, but rocksdb is looking like promising option for foundational storage system. You have things like zenfs which runs rocksdb directly on nvme storage, and then you have lot of higher level systems on top of it like Cephs BlueStore and mysql/maria MyRocks. It is not much of a stretch to imagine whole OS built around it
What is notable here is that it would be both better aligned with how the underlying storage hardware works, and simultaneously also offers more usable interface for applications.
It is pretty apparent that the biggest reason why we commonly use posixy files is inertia rather than them being very good fit for anything these days
https://en.wikipedia.org/wiki/RocksDB#Integration
https://github.com/westerndigitalcorporation/zenfs
> Why not just have the filesystem be a SQLite DB then?
One major reason not to is that there needs to be a guaranteed way to free up space, e.g. by deleting a file, or doing something similar. Currently SQLite doesn't provide any means to reliably reclaim space without allocating even more first.
Also the API is quite different, e.g. if you can write(2) a single byte at a time if you really want to, and it's relatively fast compared to doing e.g. an INSERT each time, since it goes to the temp VFS buffer first
> Why not just have the filesystem be a SQLite DB then?
A lot of people have proposed exactly this, that modern filesystems ought to be relational databases at the directory/metadata level.
It would be amazing to be able to interface with the file system using SQL and have the power of transactions.
> A: because it would suffer from all of the same problems, except worse, because it is not a filesystem
Not really. Database technologies really are far more advanced in many ways. It would not suffer from many filesystem problems, and would actually be better in many ways.
This is quite a claim to me, but only because the Vista FS died on these shoals, and because it seems we’ve had plenty of years for this idea to reach some sort of practical implementation and there’s none that’s in wide use that I’m aware of.
I think compatibility is just too much of an issue. Creating an entirely new filesystem like that, with next-generation tools around things like transactions, while keeping backwards compatibility with every quirk of every way things operate currently... it feels like an almost impossible architectural challenge. While if you just started from scratch with brand-new API's, existing applications wouldn't work with it.
Still, it doesn't stop people constantly bringing it up as a wish. Technically it would be amazing, and it wouldn't be hard to build. It just feels impossible to figure out how to migrate to.
I don’t really get it but I’m p sure it’s one of those things where I’m being a know-nothing engineer. (Been on mobile my whole life, 2009-2015 I owed primarily to SQLite-based app, p much 0 filesystem knowledge beyond the horrible analogy incoming)
The unfiltered reaction I have is “the filesystem is essentially Map<String, ByteData>, how would adding another abstraction layer _help_? How could it be impossible to migrate?” - you’ve been really informative, ignore my kvetching here, the kvetching side firmly holds if this was such a great idea _someone_ would have landed it _somewhere_ or there’d be more clear benefits stated than handwaving git and npm got it horribly wrong or more clear problems stated than “we don’t know how to migrate to it if we did it”
See also: SQLite: 35% Faster Than File System
https://sqlite.org/fasterthanfs.html
I think it might be a lot more than that, on Windows and macOS.
Edit: Yeah, 5x boost on Win10. I wish author had included XP for reference!
Edit 2: Author achieved 14x boost with a certain SQLite mode, over Windows 10.
> The third chart shows that reading blob content out of SQLite can be twice as fast as reading from individual files on disk for Mac and Android, and an amazing ten times faster for Windows.
> Why not just have the filesystem be a SQLite DB then?
Already exists: https://github.com/narumatt/sqlitefs
No idea what the performance is like, though… it uses FUSE, so that is going to add some overhead compared to an in-kernel filesystem