Well in my case I'm using llama.cpp and the mmproj is required, but I think it is just an extracted part of the original model file. Even with audio, yes it technically supports them natively and they're "encoder-free", but in practice that doesn't mean no translation or processing is required before it goes into the model. It does require much less processing though, which reduces latency.

As for coding, for sure there are many important details that a model needs to know in order to produce correctness and the smaller a model is the more it ends up training out. If there's a task you do consistently enough though, often times you can simply provide a pile of essential context so it has good enough reference to not need the extra training data.