You essentially made the libertarian argument without realizing it. According to this line of thinking, we should leave as little as possible in the hands of government exactly because it's either bad already or it will eventually be bad. We should then apply an exceptionally high bar to government responsibilities. These would be things that would be even worse in private hands (police for a simple example).
The fallacy of the libertarian argument is the assumption that it’s possible to have a small distributed government trivially doesn’t hold in presence of bigger adversarial monolithic governments elsewhere.