> Our work enhances the understanding of memorization in neural networks with practical applications towards removing it
Cool stuff. In a recent podcast Karpathy was also talking about this. He sees this as the next "target": models that don't memorise, because you can look it up in an oracle, but still keep the "reasoning" qualities.
How can you generalize without facts? They are the foundation on which generalization is built. Like programming without memorizing the keywords. Unless you make a distinction between facts that let you generalize, and facts that do not, like random ID numbers.
We want the LLM to learn the multiplication algorithm not an incomplete set of tables. The algorithm might be smaller and will be more complete.
Honestly, our technology has outpaced our epistemology. So we don't really know what a fact is or isn't. Are facts what we call our supervised learning experiences? You think the sun rises, no the earth spins. Your belief that the sun rises helps you predict sunset and sunrise. Your belief would be quaint to someone born and raised on a space station. Apollos chariot moves the sun across the sky doesn't it?