When Denmark liberalized gambling small "casinos" with slot machines and sports betting started to pop up everywhere. Those wouldn't be an issue if it was regular people popping Friday afternoon to get a coffee and spend $10 - $20 on the slot machines and perhaps put down a few bucks on this weekends big game, but you're right, it no.
The addicts and lonely line up waiting for these places to open, they spend everything they've got and the gambling places encourage it by providing them with free coffee, snacks and in some cases dinner.
No, gambling isn't immoral, but praying on the addicts, the mentally challenge and the lonely is. If you can't stay in business without exploiting the weak, you have no right to exist. The only negative consequence I see from banning gambling is the potential dangers of a black market.
And that is demonstrably very high.