There is never a good reason to permanently modify my files, if that is what is going on here. Seems like I wouldn't be able to search my photos by location reliably if that data was stripped from them.

Nothing is "permanently modifying your files".

What happens is that when an application without location permissions tries to get photos, the corresponding OS calls strip the geo location data when passing them. The original photos still have it, but the application doesn't, because it doesn't have access to your location.

This was done because most people didn't know that photos contain their location, and people got burned by stalkers and scammers.