I agree with your preference, also largely because of filters. But note that the intermediate format is Pandoc’s internal abstract syntax tree, not JSON (https://lwn.net/Articles/1064692/).

The older filter mechanism acted on a JSON serialization of the AST, but the current recommendation is to use Lua filters that work with the internal AST directly.