The perfect configuration system would be able to support any number of parameters, enforce arbitrary validation rules, be searchable, provide metadata, examples, descriptions for each parameter, provide an audit log of who/why something was changed, support rollbacks, and so on.
The problem is that folks want a flat human readable file format to solve all these problems. That's a pipe dream.
JSON and CSV are pretty close to "syntactically optimal" in terms of losslessly storing nested and tabular data, respectively, in a human readable format. We ought to just stick with those and think about how to design effective configuration systems on top of them.
Yes this ^
It totally is a pipe dream to find some magic flat human readable file that does all this, you need to think of it as a config system not a config file.
So then why aren't we questioning our core assumption here that the solution has to be the shape of a human readable file format at all and accepting "syntactically optimal" as the best we can come up with?
Other than that text files are a compatible format to work with for the tools that coders use to interact with everything (text editor/git) why does it need to be a text file at all? or even human readable? Should we even call csv human readable... ?
The perfect config system you describe could equally be implemented on top of any kind of storage (file, db, whatever) with a configuration editor ui to drive it all. And thanks to llm's even for totally one-off custom configuration files making a small web editor for them is basically free if you can give it the schema and some examples.
And ui doesn't mean it can't be keyboard driven (just to preempt people griping about using a mouse). But it does mean that you can present your config in the way that makes the most sense to actually do the configuration instead of having to work around any limitations of a syntax. An example is the configs out there with the trailing comment "foo bar #foo bar sets the the foo bars the bar in the blaze." Maybe you've configured foo 100 times, you don't need the comment to know what foo sets but the syntax demands it so you have to see it every time. In a dedicated visual config editor the comment can be pulled up only when needed.
Will your config "file" end up with some extra cruft to store things about how to show it properly in the ui? totally. but we don't care because we don't actually edit that "file" directly anymore remember?
I do think human-readability is a useful feature, to some extent. Human operators should be able to sanity-check individual components of the system for correctness, and this is so much easier if intermediate data formats are "human readable" (which I would extend as far as .sqlite, assuming a straightforward schema design).
Yeah I’m not saying you shouldn’t have a file that is possible to sanity check if that’s important to your system, but I would say (contrary to op and many comments) that we shouldn’t design around that as the core feature assumption. We don’t have trouble sanity checking our normal db tables so I’d imagine that any format that has a reliable way to see the loaded-in-memory representation would meet the same level of sanity checking.
CSV has too many features. I recently found out the hard way that line breaks are allowed inside values, so you can't actually read CSV line-by-line.
Or maybe not enough features, because there's no other way to define a linebreak. \n isn't a thing.
And also, some editors are picky about the number of commas on each line. Excel and Sheets don't care if you have uneven number of commas, but IntelliJ's CSV viewer I think just looks at the first line to determine how many columns there ought to be.
Parsing arbitrary CSV is one thing, but using text files with field- and record-delimiters is another. When used as an intermediate data format, you can often get by with naive splitting on characters (especially if you use a less common delimiter).