What you're describing is called Gell-Mann Amnesia.