This same argument is true for every bit of authority we give to law enforcement agencies (and really, the government in general). We expect they'll use those powers responsibly and within the limitations that we've ascribed, but it's always a risk that they're used irresponsibly and in situations we don't approve of.
Yes, this is an argument for not giving them more authority than necessary, but it's also an argument for holding them accountable when they do act out of bounds.
To this point, any law that gives power to government officials also needs to have explicit and painful consequences for abuse of those powers. Civilians who break the law face punishment and penalties, but government employees are almost never held to account. That needs to change.