repeat after me: yaml is not a programming language. (boo, hn strips emojis, i should know that)
very disappointed that their domain specific language is just yaml. conditionals and variable binding become insane war crimes when yaml comes to town. you have a friggin llm, do better
and when we computer folks see the word language its implied that its a computing on computers language not a friggin data format.
edit: the yaml to avoid complexity that should live in the sql side can back fire, some folks I was working with last fall were evaluating if a yaml based OLAP tool would work for them, and there were some pretty gnarly gotchas from a yaml based approach. Secondarily, theres a real case to be made that having the data linkages not visible in the charting layer means that groups of related plots with different axes wont have the right data linkage without forcing a lot more ETL for what should be a quick plot if the data already fits in memory.
what would you have chosen for your DSL? Ideally something not from scratch
i'm not sure if that caveat is sensible :)
theres actually a very important reason you want it to be an actual embedded dsl or tiny programming language!
The reason why llms can code at all is the hugeeeee amount of RL based on the loop of 1 "write code", 2 get compile time or runtime errors,3 fix it and iterate. Data file formats dont have that feedback loop so models will fall off the rails faster. Writing code that fits a latent adhoc schema just wont work as well, or will require burning a lot more context.
from that perspective, it could just be an EDSL little library in the host language, or it could be a friggin little custom language with an interpreter and good error messages.
Oh to be clear
> dct validate <file.yml>
Or any of the other commands that read the file will quickly fail with any syntax or SQL issues. It’s not just an open schema.
Fair hit on "language". I keep forgetting YAML stands for "YAML Ain't Markup Language". What we built is a declarative spec written in YAML, and that's deliberate. Think of it as HTML for boards rather than a programming language. The tradeoff is readability and structure versus flexibility.
That lack of flexibility is actually a feature for one of the bigger problems we're solving: lineage, or knowing where a number came from. If the chart layer can't transform data, the logic stays in SQL, where you can version, test and audit it.
It's also a choice about who this is for. The data community already works in SQL, Jinja and YAML every day, so there's no new language to learn.