I think the decline in UI quality is real, but I don't think the web takes all of the blame. The blame that it does take is due to a sort of mixed bag of advantages and disadvantages: web technologies make it quicker and easier to get something interactive on the screen, which is helpful in many ways. On the other hand, because it lowers the effort needed to build a UI, it encourages the building of low-effort UIs.
Other forces are to blame as well, though. In the 80s and 90s there were UI research labs in indistry that did structured testing of user interactions, measuring how well untutored users could accomplish assigned tasks with one UI design versus another, and there were UI-design teams that used the quantitative results of such tests to deign UIs that were demonstrably easier to learn and use.
I don't know whether anyone is doing this anymore, for reasons I'll metion below.
Designing for use is one thing. Designing for sales is another. For sales you want a UI to be visually appealing and approachable. You probably also want it to make the brand memorable.
For actual use you want to hit a different set of marks: you want it to be easy to learn. You want it to be easy to gradually discover and adopt more advanced features, and easy to adapt it to your preferred and developing workflow.
None of these qualities is something that you can notice in the first couple of minutes of interacting with a UI. They require extended use and familiarization before you even know whether they exist, much less how well designed they are.
I think that there has been a general movement away from design for use and toward a design for sales. I think that's perfectly understandable, but tragic. Understandable because if something doesn't sell then it doesn't matter what its features are. Tragic because optimizing for sales doesn't necessarily make a product better for use.
If a large company is making a utility cares, they'll have a ux person/s, sometimes part of a design team, to make sure things are usable.
But if you're really big, you could also test in production with ab testing. But as you said, the motivation tends to be to get people to click some button that creates revenue for the company. (subscribe, buy, click ad)
Somewhat related to this, the google aistudio interface was really pushing gdrive. I think they reduced it now, but in the beginning if you wanted to just upload a single file, you had to upload it to gdrive first and then use it.
There was also some annoying banner you couldn't remove above the prompt input that tried to get you to connect to gdrive.
Yes true. It's basically form over function and it's not just limited to Web UIs.
Windows 11, iOS7, iOS26 are just some example of non Web UIs, which focused first on optimizing for sales, i.e. making something look good without thinking about usability implications.
Fortunately usability testing is still pretty much a thing. Good articles here: https://www.nngroup.com/search/?q=usability+testing