I think if you're a citizen, then I agree with you. However if you're an alien (legal or not) I think they should be allowed to figure out where you are.

They absolutely should, but the onus is on them to figure out a way to do that within the confines of existing law. Existing law notably does not make ICE or CBP a super-agency that can do whatever it wants.

I think data brokers are legal. I think purchasing data is legal. So what is the problem?