I think the point is exactly that just because someone is smart in one area doesn't mean they know anything about other areas, and yet we constantly assume that's true.

We probably shouldn't be treating Nobel prize winners as the smartest people generally, just specifically in the field of their discovery. Very common fallacy to fall for though, and when everyone treats you like a general genius, of course you're going to give your opinions.