Any multimedia project trying to support a large number of formats, whose usage in the wild differs by orders of magnitude, is going to have code of varying quality (although quality is not strictly correlated with usage: age and complexity are also big factors, among others). GStreamer puts plugins into different categories (-good, -bad, etc.) based on things like the maturity of the code, which helps you judge what risks you are taking. With FFmpeg it is harder to know which formats are more likely to have issues. Of course GStreamer can use FFmpeg, in which case you will also have all of FFmpeg's problems.

In both cases you are best off restricting things to what you actually use.