To clarify, it depends on where you are. In the US, federal land comes in different tiers of protection. National Parks are tourist destinations, generally requiring you to stay on trail, take nothing, leave nothing, etc. National Wilderness Areas allow you off trail, but no motor vehicles. National Forests allow a pretty varied level of recreational usage, often including non-commercial rockhounding, and may allow off-road vehicles (depends on the forest), and then Bureau of Land Management (BLM) land is basically a free-for-all, allowing pretty much any recreational activity and even commercial activity with the appropriate permits.

Out here in the west, you'll find a LOT of land in all 4 categories, not to mention state-level parks, which at least here in California allow some rockhounding near streams and beaches.

Is it really so bad? My son and I have hiked out to fairly remote places and dug up some rocks to find fossils. I know there are more there and very few people are likely to visit the spot, let alone to find fossils in the dirt. We leave very little trace at all. Is this actually frowned upon?

Stolen? I have mining claims. What's stolen? To boot, these are in areas nobody hikes because it's too dangerous for hikers. These are in actual mining districts.