> I just hate the argument that government inherently because it's government is inefficient and will never be effective
The more general argument is that monopolies are inefficient, and most instances of government programs are monopolies.
> In the US half the people in charge of the government hate it and want it to fail.
This is uncharitable. It's more accurate to say that they expect it to fail and hate government programs because they regard them as inefficient and net harmful on average.
> If everyone agreed that, once we make a choice on what the government is going to do we should do it well, we could have government services that are just as effective as whatever private enterprise example people always like to cite
The inefficiency is caused by the principal-agent problem more than anybody's ideology. Even if every politician wanted and expected the government to be effective, you still have the problem where a government program is providing a service consumed by e.g. 5% of the population and is serving 20% of them poorly. This is only 1% of the voters, who themselves have to balance their vote against their positions on every other issue on the ballot, whereas in a competitive market the company doing that would be losing 20% of their customers to a competitor and a competitor would be available to satisfy that 20% of the customers.
Meanwhile if a program is inefficient but its total cost is <1% of the total government budget, nobody is paying attention to it, but the same is true of most other programs which causes the average efficiency to be poor. Whereas the proprietor of a small business which is <1% of the economy is going to care quite a lot if it's wasting money and be paying attention to that business in particular, because it's their business and it's their money.
We don't have great solutions for these in the public sector. The best you can say is that sometimes it's worth it. For example, every taxpayer can use tax filing software and it has no unit cost, so even if the government is less efficient than a private company, that could still be worth it because you can spread the fixed cost of developing it over so many people. This is a general argument for the government to create and publish open source software whenever there is a reasonable expectation it would be widely used.
What it isn't is a general argument for government programs, because products with zero unit cost are relatively uncommon and so are software products that are both widely used (as opposed to having a particular niche) but not already well-served by existing free software. Software in the service of regulatory compliance is unusually well-suited to being published by the government. (And they should definitely be publishing the source code.)
> The more general argument is that monopolies are inefficient, and most instances of government programs are monopolies.
Even more general, markets where it is hard for new competitors to enter are inefficient. It's perfectly possible for a market with 10 or 20 competitors to stop competing for mutual benefit. I'd even argue a monopoly (a single dominant company) is fine, so long as it's easy for competitors to enter the market.
Consider the WW1 Christmas truce and other fraternizations. If the same actors are together long enough, they'll act for mutual ease. That was a good thing in WW1, but it's not a good thing when companies stop competing in a capitalist society.
> It's perfectly possible for a market with 10 or 20 competitors to stop competing for mutual benefit.
As I remember it, the empirical research says two competitors can keep a stable monopoly cartel going for a long time. three can maybe do it for a while. With larger numbers things get very unstable.
Source? I would enjoy reading more about this. I wonder if this was a simulation or an observational study of real markets?
> I wonder if this was a simulation or an observational study of real markets?
It's kind of both.
Try to name a market where the largest company in the market has <15% market share but they still have a successful cartel going. They exist but you're going to keep finding that their enforcement mechanism is some kind of violence or (equivalently, through the government's monopoly on violence) regulatory capture.
Drug cartels can have lots of members because they murder you for defecting. Landlords collude through zoning boards. OPEC is literally a cartel of governments.
You can also kind of reason this out.
Suppose a market has two companies and the smaller one has 40% market share. If they could increase their market share to 60% by cutting their margins in half, that's losing money, so screw that. The bigger company doesn't even have to buy them off.
Suppose a market has eleven companies and the smallest one has 1% market share. If they could increase their market share to 20% by cutting their margins in half, they're making 10 times more than they do now. The other companies would have to buy them off. But how? More market share without lowering prices? Even if they could arrange it, any company with less than 10% market share would rather have 20% at half the margins, but in a market of eleven the smallest company can't have more than 9% without making someone else the smallest company. And that's if having the lowest price only gets you 20% market share and not more.
Now, you can make up numbers that allow coordination with arbitrarily many companies. Maybe lowering your margins to zero doesn't get you any more customers at all and then no one would have any reason to do it. But how many real markets work like that?
I'm not an economist, but I read a lot of what they write, and it seems like an established fact in the field.