> I think so. This machine might produce answers we value, but it would not, in itself, produce human understanding of those answers.
It's nice that the author is optimistic, but won't the AI be best placed to dumb down its increasingly complex proofs into a language us lowly humans can understand? To keep thinking until it can refactor complex proofs into ones from 'the book'?
As they go on to explain, a human understandable proof is different than a human actually understanding the proof. That actual human understanding (like, in a brain of a human) is one of their stated goals.
Producing human understandable proofs is possibly a job best for humans today, but the author appears to agree with you that this is probably fleeting (and argues that even if you disagree, it should probably be treated as if it is fleeting when planning for the future):
> Right now AI systems arguably underperform us at theory-building, asking questions, exposition, … so we could prioritize and reward those skills. I think this is unwise: compare the speed at which the academy adapts to the speed at which model capabilities improve. We need to consider the endgame. If the models remain incapable in some domain, we can adjust later.
AI will never be able to dumb down a proof to a level simple enough for someone to understand who has never studied math and put a lot of effort into it. Some concepts just need time and effort to absorb no matter how relatively simply they are phrased.
Sure, but some proofs are so complex that even experts can't follow them. As AI progresses, this may constitute an increasing fraction of proofs. I meant that AI could work to find the simplest possible proof.