It's mixed bag... the designer of a given website has an intended look/feel and style... if you override that you can do as you like, but it's not like the author's intent should always simply be dismissed.
Beyond this, not every web developer expressly wants to burden a browser to a specific web font payload when they have a close/suitable match where this modern font stack is good enough in terms of design intent.
Third, if all else fails, the user sees their own selected default... I'm not sure that I understand the objection here... As long as appropriate semantic markup and the font is one that actually scales to appropriate px/pt then it should be fine. If the selected font/typeface doesn't, then it's on the user to select a better default/fallback.
> it's not like the author's intent should always simply be dismissed.
Yes it is. The designer should always understand that the user is ultimately in control of a web page, and that their (the designer's) vision is not what matters at the end of the day.
If you choose to use w3m or lynx you get what you get. Same for disabling fonts or JS... most people don't have time to cater to 0.05% of users who go way off the norm.