To clarify, it depends on where you are. In the US, federal land comes in different tiers of protection. National Parks are tourist destinations, generally requiring you to stay on trail, take nothing, leave nothing, etc. National Wilderness Areas allow you off trail, but no motor vehicles. National Forests allow a pretty varied level of recreational usage, often including non-commercial rockhounding, and may allow off-road vehicles (depends on the forest), and then Bureau of Land Management (BLM) land is basically a free-for-all, allowing pretty much any recreational activity and even commercial activity with the appropriate permits.

Out here in the west, you'll find a LOT of land in all 4 categories, not to mention state-level parks, which at least here in California allow some rockhounding near streams and beaches.