Those decoders aren't even compiled and activated in the released binaries. But in any case, why would that be FFMPEGs problem?

Please stop spreading this misinformation. At least in Debian this is enabled by default (and as another post indicates, Ubuntu as well).

Run the following command to confirm:

ffmpeg -codecs|grep sanm

If you're using ffmpeg it's recommended to just enable the things you need, or only accept some container formats. But yes, in a generic package everything is enabled.