The problem isn't just whether they're hiring below market. It's that market rates themselves are impacted by there being a whole underclass of workers who can't just freely move between jobs, and has to speedrun getting a new one if they ever lose their existing ones. The precarious nature of it makes an H1-B worker less likely to speak up about most things related to working conditions due to the precarious nature of the job. And that impacts everyone else -- because the fastest way to get a big raise is by getting a new job, so there's less incentive to pay people better if fewer workers can actually move around and you can threaten the ones that can purely through the existence of ones that can't.
I don't think the solution is just getting rid of them all -- actually a longer grace period (say, a year?) would make it easier to leave abusive employers. But it should include more scrutiny on whether employers really can't find local hires in the first place, or just aren't willing to train them or work with people that have even slightly more bargaining power.
Abuse of the H1-B program is broader than just hiring people for less money right now. It's also about hiring people who won't ask for more money over a longer timeline. It's about having a whole class of workers that is uniquely vulnerable to terrorist bosses, which then gives those employers more leverage over everyone else. And it's about the large-scale market manipulation that comes with having this easily accessible.