also worth noting that Vega-Lite is literally just fucking that and AI already does good job with producing JSONs for it

I work with someone who did a lot of work with this to improve our ability to generate awesome visualizations with little thinking. It's a very powerful language but needs guardrails and guidance, particularly if you want end users to be able to produce consistent and standardized visualizations without knowing anything about it.

Also guidelines sent to the agent may or may not get ignored if they are just part of the context :(

A little bit more than that! Here shows why a higher-level abstraction can be helpful for chart generation: https://github.com/microsoft/flint-chart#features