Yup, if you're using OpenCV for instance compiling instead of using pre-built binaries can result in 10x or more speed-ups once you take into account avx/threading/math/blas-libraries etc...

Yup. The irony is that the packages which are difficult to build are the ones that most benefit from custom builds.