Let's be honest, the real 1999 design would have been a custom binary format that happens to use the endian of the system it was developed on, and leaks bits and pieces of unflushed memory buffers whenever it writes to disk.
1999 is exactly right for the start of the XML hype, everything had to be XML, it would single handedly solve the software crisis (after OOP failed to do that) because everything would be able to talk to everything!
The whole Italian e-invoicing system is based on SOAP XML, and it's atrocious. They approved the specs in 2013, so it was already dated when it came out.
SQLar[1], see [2] for some reasoning around why a database is preferred over zipped XML. Though, I'd be fine with a DBM-style database too as we only need the key-value part of it.
The problem with using database blobs for load/save is that you usually need a full database client in the application. SQlite advertises that use case, but it is complete overkill. You never need to run any sort of complex SQL query on an image file format for instance. Using XML+ZIP in this day and age is also a strange decision, but at least that way the data is inspectable with unzip, a text editor and an image viewer (assuming they use a standard image format to store the raw pixel data).
Think of Lightroom or automation over files. Many semi professionals from wedding photographers to designers want some form of batch automation and organization system over their files.
Several megabytes/gigabytes assets and you want to extract metadata, a preview, running as a batch some filter/compression/, conversion to CMYK, text injection ... fast partial read/write access would be nice.
Right now, most reads are performed through indexes because those files are slow to read.
If we take 10k sqlite files and want to retrieve a row, we would be around 3s on SSD, maintaining preemptive indexes become less important for a lot of use cases.
Change management and versioning also becomes quite efficient - sqlite can be configured to not offset bytes, so CVS like Epic Lore can efficiently delta the files and store minimal delta, or the file format itself can keep its edit history. Oh and it's 3x-10x less bytes without compressing the whole thing, so pages are stable through time.
About needing SQLite client, it's real but it's roughly the same size as an XML parser.
> About needing SQLite client, it's real but it's roughly the same size as an XML parser.
what I like about it being XML is I can just open it with any text tool and inspect it. It's human readable so I can edit and debug it manually, no need to have a parser or an extra application just to see what's in my file
Sqlite is probably overkill, you will probably never have actual relational data in an image format, But what it does bring to the table is a built in b-tree based storage, that is, you don't need to load the entire file into memory to edit it, in a prior age we would use Berkeley db for this. Sqlite in this role(a file format) is probably best thought of as a better superset of the berkleydb style key value store, more than one table per file and additional columns/indexes to keep metadata in.
Nothing wrong with XML, it is well understood, and the tooling is pretty good. But partial loads/edits is one thing it can not do.
> You never need to run any sort of complex SQL query on an image file format for instance.
Sure you will. Plenty of features that don't exist, or are implemented badly, because you can't easily do it.
Quick mental translation table: if you think "iterate over every ..." or a `for` loop, that's your SELECT query. If you think about `if` conditions, that's the parts that go after FROM clause.
In order to have any advantage from this, you would have the added complexity of splitting your file format into tables that can be queried in a useful manner. However, for an image file format, you most likely need to hold the entire definition in memory at all times anyway. Assuming that’s the case, doesn’t XPath get you there most of the way (assuming XML), with _way_ less complexity?
Image data is just binary blobs. You aren't splitting that into channel columns or anything. But an image file for an editor like Gimp isn't one image blob. It's dozens or hundreds of them - one or more per layer - along with tons of associated metadata at every level.
All that tends to fit sensible schemas and managing it is what SQLite shines at.
I don’t see how this addresses my point that zipped XML gives you the same thing, but simpler. I understand that a GIMP file is many images, so that makes a zip feel like a great fit to me. The only advantage I see for using a full-blown DB is ensuring consistency with references, which admittedly is a plus. But beyond that, what do you gain?
- Continuously parsing and writing and reparsing text, 90% of which is useless (that's the JSON/S-expressions vs XML argument)
- Forcing a diverse relational structure to fit a tree hierarchy, hand-writing all the logic that manages representation change - either explicitly, at serialization boundary, or implicitly, in every single access operation you're doing to refer to some data;
- Or worse, using an off-the-shelf, generic object/XML mapper, in which case you just compound the bloat even more.
SQLite is one of the single most battle-tested and ubiquitous piece of software in the history of mankind. Anything "simpler" you're going to pick up is much more likely to be buggy and broken, and will definitely be orders of magnitude slower.
SQLite has it's own issues - for example it doesn't support checksums. Quite bizarre for a database file format where integrity is supposedly highly valued. Saying it's the job of the filesystem doesn't help when all major OSes don't enable checksums in their default filesystem.
With a zip file at least you know your file was corrupted.
SQLite is fast, but it won't be faster than a hot loop in C. Having the loop construct dictated by the file format seems bad. For images it seems more reasonable to have them in-memory, except for huge image edge cases.
Images are the red herring. Pixel data is best read in hot loops in C, but that would be stored as blobs in SQLite anyway.
It's all the metadata around the image that's interesting. Images have layers, dozens or hundreds of them (this literally scales with how good your software is at handling those - the faster, and more powerful layer UX is, the more they get used). Some are pixel layers, other are effect layers, text layers, vector layers. Layers have metadata - names, sizes, colors, tags, types, special effects, and a bunch of other stuff I don't know because I don't use that 80% of features of GIMP/Photoshop/Affinity.
Then you have document level metadata, UI-specific metadata, etc. Also undo history. A lot of that is relevant to the work on images themselves, and changes in realtime, and can get even more useful if querying it wasn't such a PITA.
That - not the binary pixel blobs - is the selling case of using SQLite as application data format.
the SQLite archive format (it's probably worth linking to the main documentation[1], instead of the very old experimental repository) may not really be a good fit for something like GIMP's native file format. (To be clear, "SQLite archives" are not special compared to any other database: it's just a well-defined schema for an sqlar table, which the sqlite3 command line tool is able to create, update, and extract using syntax like the tar command.)
That being said, SQLite would still be a good choice, especially as it's a format that's really intended to be modified in-place, and has good data integrity features (eg: keep WAL enabled so that mid-save crashes/shutdowns don't corrupt your file), neither of which are provided by Zip. You could even just run zlib on data (be it XML or what have you) if optimizing the on-disk size of the file is desirable.
Yes, this is not too rare to see base64 images in JSON but I won't recommend that.
You gain back most of the base64 overhead when you compress it though. It's slower but probably often worth it if the alternative is few more async HTTP queries that you would only fetch once.
To answer seriously, my parent comment is a joke, JSON is a simpler format that maps better to most programming languages internal memory representations. Developers tend to prefer JSON’s simplicity over XML.
> most programming languages internal memory representations
Often heard wrt JSON but incorrect. It maps to the primitive types in JavaScript. But almost all programming languages treat floats and integers different, make distinction between char and strings and many have some form of date/time. JSON has neither.
In that direction, XML is much closer since every node is a triple (name, value, attributes) so can have type info, json is a tuple. And Protobuf, while not popular, gets this completely right.
I think it's too risky to treat numbers in JSON as something else than IEEE754 64bits floats. But yes, JSON is small and doesn't do datetimes, char, comments, and a million other things XML does.
But you don't need to think much about memory representation when you parse a JSON, and the developer experience is a lot more pleasing than browsing a XML tree. That what used to matter.
If this is intended for reading and writing by humans, then JSON doesn't cut it since you don't get comments (and perhaps also because of the weirdness of 64-bit floating-point values). Plus, XML is more "structurally nuanced". I suppose JSON is simpler to parse, but it's a trade-off of features, it's not like one is bad and the other is good.
Thanks! I had the exact same question for various of these posts here. I know these may be different groups of people, but I always see people advocating for simplicity, and zipped-XML is as simple as it gets, needs barely any extra dependencies (none in GIMP I assume), and is proven to work well (Word etc).
I also don’t understand why SQLite would be preferable here, considering you will likely also store large binary files alongside your document definition.. What does a db engine give me here?
The obvious alternative is SQLite, and it has many advantages. If you want to do a "zipped list of files", SQLite does that just fine (that's what SQLar is), but it can do so much richer data. Even if you don't want that, it still offers resiliency that "zipped XML" can't match: if your software or computer crashes in the middle of saving your file, it'll almost certainly corrupt it. With SQLite, not an issue: all transactions are atomic, they either happen entirely or not at all.
I don't know what "mature tooling" you're talking about for zipped XML, but I guarantee you it's not going to be better (or more mature) than SQLite and its ecosystem.
The problem was that people hyped it up as the solution for every problem and all the world's ills, so it wound up being used where it had no business being used (and therefore badly).
XML was a hammer used to bang in a lot of things that weren't remotely nail-shaped, and accordingly, a certain percentage of traumatised people despise it and react to any mention of it with fear and loathing.
It's kind of neat when you have an application for it that really leans into its strengths.
Let's be honest, the real 1999 design would have been a custom binary format that happens to use the endian of the system it was developed on, and leaks bits and pieces of unflushed memory buffers whenever it writes to disk.
No that would be an early 1990s file format ;)
1999 is exactly right for the start of the XML hype, everything had to be XML, it would single handedly solve the software crisis (after OOP failed to do that) because everything would be able to talk to everything!
Indeed, early 00s was still the SOAP XML era if I remember right, WSD, SOA, ...
SOAP XML is so uselessly convoluted.
The whole Italian e-invoicing system is based on SOAP XML, and it's atrocious. They approved the specs in 2013, so it was already dated when it came out.
Jar
What would you consider good modern design?
SQLar[1], see [2] for some reasoning around why a database is preferred over zipped XML. Though, I'd be fine with a DBM-style database too as we only need the key-value part of it.
[1] https://sqlite.org/sqlar/doc/trunk/README.md
[2] https://www.sqlite.org/affcase1.html
The problem with using database blobs for load/save is that you usually need a full database client in the application. SQlite advertises that use case, but it is complete overkill. You never need to run any sort of complex SQL query on an image file format for instance. Using XML+ZIP in this day and age is also a strange decision, but at least that way the data is inspectable with unzip, a text editor and an image viewer (assuming they use a standard image format to store the raw pixel data).
Think of Lightroom or automation over files. Many semi professionals from wedding photographers to designers want some form of batch automation and organization system over their files.
Several megabytes/gigabytes assets and you want to extract metadata, a preview, running as a batch some filter/compression/, conversion to CMYK, text injection ... fast partial read/write access would be nice. Right now, most reads are performed through indexes because those files are slow to read.
If we take 10k sqlite files and want to retrieve a row, we would be around 3s on SSD, maintaining preemptive indexes become less important for a lot of use cases.
Change management and versioning also becomes quite efficient - sqlite can be configured to not offset bytes, so CVS like Epic Lore can efficiently delta the files and store minimal delta, or the file format itself can keep its edit history. Oh and it's 3x-10x less bytes without compressing the whole thing, so pages are stable through time.
About needing SQLite client, it's real but it's roughly the same size as an XML parser.
> About needing SQLite client, it's real but it's roughly the same size as an XML parser.
what I like about it being XML is I can just open it with any text tool and inspect it. It's human readable so I can edit and debug it manually, no need to have a parser or an extra application just to see what's in my file
That's such an unbelievably niche use case for an image file format, though. Extremely the wrong thing to optimize for.
Sqlite is probably overkill, you will probably never have actual relational data in an image format, But what it does bring to the table is a built in b-tree based storage, that is, you don't need to load the entire file into memory to edit it, in a prior age we would use Berkeley db for this. Sqlite in this role(a file format) is probably best thought of as a better superset of the berkleydb style key value store, more than one table per file and additional columns/indexes to keep metadata in.
Nothing wrong with XML, it is well understood, and the tooling is pretty good. But partial loads/edits is one thing it can not do.
> You never need to run any sort of complex SQL query on an image file format for instance.
Sure you will. Plenty of features that don't exist, or are implemented badly, because you can't easily do it.
Quick mental translation table: if you think "iterate over every ..." or a `for` loop, that's your SELECT query. If you think about `if` conditions, that's the parts that go after FROM clause.
In order to have any advantage from this, you would have the added complexity of splitting your file format into tables that can be queried in a useful manner. However, for an image file format, you most likely need to hold the entire definition in memory at all times anyway. Assuming that’s the case, doesn’t XPath get you there most of the way (assuming XML), with _way_ less complexity?
Image data is just binary blobs. You aren't splitting that into channel columns or anything. But an image file for an editor like Gimp isn't one image blob. It's dozens or hundreds of them - one or more per layer - along with tons of associated metadata at every level.
All that tends to fit sensible schemas and managing it is what SQLite shines at.
I don’t see how this addresses my point that zipped XML gives you the same thing, but simpler. I understand that a GIMP file is many images, so that makes a zip feel like a great fit to me. The only advantage I see for using a full-blown DB is ensuring consistency with references, which admittedly is a plus. But beyond that, what do you gain?
You're not:
- Continuously parsing and writing and reparsing text, 90% of which is useless (that's the JSON/S-expressions vs XML argument)
- Forcing a diverse relational structure to fit a tree hierarchy, hand-writing all the logic that manages representation change - either explicitly, at serialization boundary, or implicitly, in every single access operation you're doing to refer to some data;
- Or worse, using an off-the-shelf, generic object/XML mapper, in which case you just compound the bloat even more.
SQLite is one of the single most battle-tested and ubiquitous piece of software in the history of mankind. Anything "simpler" you're going to pick up is much more likely to be buggy and broken, and will definitely be orders of magnitude slower.
SQLite has it's own issues - for example it doesn't support checksums. Quite bizarre for a database file format where integrity is supposedly highly valued. Saying it's the job of the filesystem doesn't help when all major OSes don't enable checksums in their default filesystem.
With a zip file at least you know your file was corrupted.
https://avi.im/blag/2024/sqlite-bit-flip/
SQLite is fast, but it won't be faster than a hot loop in C. Having the loop construct dictated by the file format seems bad. For images it seems more reasonable to have them in-memory, except for huge image edge cases.
Images are the red herring. Pixel data is best read in hot loops in C, but that would be stored as blobs in SQLite anyway.
It's all the metadata around the image that's interesting. Images have layers, dozens or hundreds of them (this literally scales with how good your software is at handling those - the faster, and more powerful layer UX is, the more they get used). Some are pixel layers, other are effect layers, text layers, vector layers. Layers have metadata - names, sizes, colors, tags, types, special effects, and a bunch of other stuff I don't know because I don't use that 80% of features of GIMP/Photoshop/Affinity.
Then you have document level metadata, UI-specific metadata, etc. Also undo history. A lot of that is relevant to the work on images themselves, and changes in realtime, and can get even more useful if querying it wasn't such a PITA.
That - not the binary pixel blobs - is the selling case of using SQLite as application data format.
> Using XML+ZIP in this day and age is also a strange decision
I agree with you that SQlite is overkill but honestly curious to know why you think xml+zip is strange? what would you use instead?
the SQLite archive format (it's probably worth linking to the main documentation[1], instead of the very old experimental repository) may not really be a good fit for something like GIMP's native file format. (To be clear, "SQLite archives" are not special compared to any other database: it's just a well-defined schema for an sqlar table, which the sqlite3 command line tool is able to create, update, and extract using syntax like the tar command.)
That being said, SQLite would still be a good choice, especially as it's a format that's really intended to be modified in-place, and has good data integrity features (eg: keep WAL enabled so that mid-save crashes/shutdowns don't corrupt your file), neither of which are provided by Zip. You could even just run zlib on data (be it XML or what have you) if optimizing the on-disk size of the file is desirable.
[1] https://sqlite.org/cli.html#sqlite_archive_support
Isn't the implementation the spec for SQLite?
Not really a great option for an image format, where we therefore can't have multiple implementations. Unless I misremembered.
SQLite file format spec: <https://sqlite.org/fileformat.html>
Thank you for the prompt clarification!
Compressed JSON with the binary content encoded in base64 strings, obviously.
So you compress a format that inflates binaries with 30%?
That not only ends up larger than "just the binary", it also eats a lot of extra CPU to (de)compress AND encode-decode.
This idea is novel, but wasteful.
(edit: I thought you were serious, so I answered serious. You were not ;)
Yes, this is not too rare to see base64 images in JSON but I won't recommend that.
You gain back most of the base64 overhead when you compress it though. It's slower but probably often worth it if the alternative is few more async HTTP queries that you would only fetch once.
Why would zipped JSON be fundamentally superior to zipped XML?
To answer seriously, my parent comment is a joke, JSON is a simpler format that maps better to most programming languages internal memory representations. Developers tend to prefer JSON’s simplicity over XML.
> most programming languages internal memory representations
Often heard wrt JSON but incorrect. It maps to the primitive types in JavaScript. But almost all programming languages treat floats and integers different, make distinction between char and strings and many have some form of date/time. JSON has neither.
In that direction, XML is much closer since every node is a triple (name, value, attributes) so can have type info, json is a tuple. And Protobuf, while not popular, gets this completely right.
I think it's too risky to treat numbers in JSON as something else than IEEE754 64bits floats. But yes, JSON is small and doesn't do datetimes, char, comments, and a million other things XML does.
But you don't need to think much about memory representation when you parse a JSON, and the developer experience is a lot more pleasing than browsing a XML tree. That what used to matter.
If this is intended for reading and writing by humans, then JSON doesn't cut it since you don't get comments (and perhaps also because of the weirdness of 64-bit floating-point values). Plus, XML is more "structurally nuanced". I suppose JSON is simpler to parse, but it's a trade-off of features, it's not like one is bad and the other is good.
Welcome to 2026 unfounded hot takes.
Or less snarky: what's your gripe with zipped XML? It compresses reasonably well, has a useful structure, and has decades of mature tooling around it.
Thanks! I had the exact same question for various of these posts here. I know these may be different groups of people, but I always see people advocating for simplicity, and zipped-XML is as simple as it gets, needs barely any extra dependencies (none in GIMP I assume), and is proven to work well (Word etc).
I also don’t understand why SQLite would be preferable here, considering you will likely also store large binary files alongside your document definition.. What does a db engine give me here?
The obvious alternative is SQLite, and it has many advantages. If you want to do a "zipped list of files", SQLite does that just fine (that's what SQLar is), but it can do so much richer data. Even if you don't want that, it still offers resiliency that "zipped XML" can't match: if your software or computer crashes in the middle of saving your file, it'll almost certainly corrupt it. With SQLite, not an issue: all transactions are atomic, they either happen entirely or not at all.
I don't know what "mature tooling" you're talking about for zipped XML, but I guarantee you it's not going to be better (or more mature) than SQLite and its ecosystem.
Why XML bad?
Fundamentally, it's not.
The problem was that people hyped it up as the solution for every problem and all the world's ills, so it wound up being used where it had no business being used (and therefore badly).
XML was a hammer used to bang in a lot of things that weren't remotely nail-shaped, and accordingly, a certain percentage of traumatised people despise it and react to any mention of it with fear and loathing.
It's kind of neat when you have an application for it that really leans into its strengths.