Tao's central point seems to be:
"In short, the indiscriminate use of powerful solution-extraction tools can achieve the immediate short-term goal of solving problems at hand, but at the cost of sustaining the ecosystem for the next wave of progress, or in understanding the progress already obtained. "
I am no mathematician, may have misunderstood his point and would be delighted to receive any corrections.
I definitely see what he is saying.
In my work as a graphics programmer I often find that I look at a problem and will immediately see how to solve it, more or less. But the devil is in the details and often nothing works unless you get every detail right. So you spend a lot of time coming up with complex solutions, then boiling them down to simpler versions. In the end you often end up with a fix which is short, simple, and seems obvious. But it gets a lot of subtle details just right and avoids countless potential issues you wouldn't know if you hadn't failed a lot getting there.
And that is actually how you learn and master the craft.
Now, imagine you describe how you sort of solve it to a machine and it spits out the simple, correct implementation and you nod approvingly, never knowing all the ways it could have gone wrong. If this is how mathematics - or programming - is done from now on, no one will actually master their craft. I definitely see why this would worry someone whose career is built on mastery of the craft and a legacy meant to teach the next generation.
Doesn't this seem to be where all domains are headed?
I get kinda freaked out when I feel like all the AI "utopianists" haven't taken the next logical step of thinking about what society looks like when humans are subpar in every domain (and you may argue this won't happen, though I'm becoming more and more a believer that it will, but my point is the utopianists believe that this absolutely will happen, and that it's also a wonderful thing). How motivated do you think folks will be to do the hard cognitive work to focus on things like math problems when there is a good chance AI will do it better?
Doesn't the premise that there's something inferior about these AI solutions, imply that there is something superior about human intelligence and that there will continue to be some kind of useful work for humans to do?
> Doesn't the premise that there's something inferior about these AI solutions, imply that there is something superior about human intelligence and that there will continue to be some kind of useful work for humans to do?
Not necessarily. The "something superior about human intelligence" may have dependencies that "these AI solutions" are able to eliminate, such as the motivation to refine intellectual talent to a high level. Basically, AI could kick the ladder out from under human intelligence but be incapable of actually surpassing it in important ways, enabling a burst of advancement that's also a dead end. Sort of like https://en.wikipedia.org/wiki/The_Road_Not_Taken_(short_stor....
So the AI could be inferior but there's still no useful work for humans, because the environment doesn't allow them to work up to that level anymore.
This is kinda feeling a bit like SBF's coin flip bet: https://www.businessinsider.com/sam-bankman-fried-coin-flip-.... Achieve human-superior AGI this generation or humanity stagnates.
I don't think humans necessarily become subpar when AI can do most of the work. Taking my own personal example, I have far more intellectual curiosity and improved my skills in programming far more with Claude Code than for 15 years of programming without AI simply because I was bogged down by boilerplate and grunt work. Now that AI handles most of the boilerplate and grunt work and can handle harder and harder problems, I have the time and space to work on unexplored frontier problems.
So, no, my skills have not become subpar, but have only become stronger because of the presence of AI.
So when AI is also better at coming up with the tasks and already solving the frontier problems? 1M AI agents running full time thinking at 10M times faster than humans, with 100 times the intelligence in each agent? Any human ideas regarding "frontier problems" or "what the AI should focus on" are irrelevant. We have automated our own thinking and human cognition will be about as important as being "the best at abacus". Its an interesting party trick, but in no way useful.
Before AI, did you find that the existence of human expertise in the world far surpassing yours 'automated your thinking and cognition' or limited your intellectual scope to inventing party-tricks? I've always found greater intelligences than mine are like runways for curiosity. It's hard to believe anyone in their presence would fall into a stupor instead of being highly stimulated by them - and I can't see why it would be any different with ASI. Actually, this "AI undoes your cerebral strapping" argument seems exactly like the kind of slovenly thinking it wants to blame on AI.
yes to a degree. When I was 16, my idea about "what cutting edge thing to work on" was pointless, I couldn't make a meaningful contribution. Most developer's work is mundane and wrote. Things they think are "cutting edge" are actually just ideas they haven't considered, or markets they don't realize are saturated with really good players already. It takes time, effort, and knowledge to make a meaningful contribution. But if those fields are saturated by players that are beyond human ability, then its a bit like trying to become a competitive chess player when the bar for entry is stockfish -- no human will ever again pass this bar. At a certain point, the bar for being paid for intellectual contribution will surpass human abilities. The things humans can understand and ideas they contribute will not be worth much.
To make an unrelated anology, the presence of Lebron James definitely dampens my ability to contribute meaningfully to basketball. Even with all my practice, I cannot achieve his level. But if he didn't exist, and the best 90% of basketball players were eliminated, then I might be able to meaningfully contribute. Of course, I can still play basketball as a hobby, but you were talking about making meaningful contributions on hard novel problems at the frontier of knowledge and ability. That frontier is moving away from humans and will soon be the domain of electronic experts.
I think the main point is the ubiquity of AI intelligence. If there is a human far surpassing me, but unavailable to me, then I have to work through it myself. If the answer is a prompt away, then why bother.
> what society looks like when humans are subpar in every domain
> when AI can do most of the work
You're not really responding to the core hypothetical of his comment
To me it reads as that: for utopians, you may benefit from LLMs now, but they'll still surpass you later, what then ?
Thank you for putting it so succinctly. I find that lots of people get "stuck" in the current state of things - they think that AI will continue to improve, but they inherently believe (perhaps subconsciously) that humans will always have the edge in things like "ingenuity", "inventiveness" or "curiosity".
I don't blame them, I also believed that until very recently. I thought there was something unique and special about sentience and consciousness, but my thinking has changed in that I don't think machines need to develop the sentience we experience to still surpass us cognitively along all dimensions.
But iff you really examine what grounds your beliefs, and then consider that AI very well could surpass humans even in domains like curiosity and real creativity (at several orders of magnitude faster), it starts to look like a very, very uncomfortable future for humans IMO.
I saw this quote on Twitter:
> Best outcome: we become pets
> Worst outcome: we become cows
I don't get this take at all. Do you feel inferior that bears are stronger than you? Or that a machine can lift more weight than you? Or that a calculator can calculate faster than you?
These AI things are machines which are completely under our control. Even if they become more intelligent than us, that doesn't mean that it has authority over us. There are plenty of very intelligent humans which work under the authority of much less intelligent humans. Many engineers are smarter than their CEOs.
What makes you think that just because an AI is more intelligent that it will not be completely subject to our authority? It's just a tool in our hands.
> What makes you think that just because an AI is more intelligent that it will not be completely subject to our authority? It's just a tool in our hands.
Because it would be like 100 chimps trying to make a million humans subject to their authority.
Geoffrey Hinton, considered "The Godfather of AI" for his pioneering work on neural networks, and who now says he somewhat regrets his life work due to the existential risks, has said similar things, e.g. "How many examples do you know of a more intelligent thing being controlled by a less intelligent thing? There are very few examples"
So, what if they surpass me? It's still a tool under my control. Many engineers work under the authority of a CEO, even though they're smarter than them. Just because AI might be more intelligent doesn't mean it's still not a machine that is completely under my control.
> I have the time and space to work on unexplored frontier problems.
Like what?
Mostly low-level performance optimization and new programming language design.
Obviously it is a ludicrous example, but there are books for this: The Culture novels by Iain. M. Banks.
I'd be quite happy as a citizen of The Culture.
But, on a more serious note, or more realistic, I find it vaguely insulting that you think that so many of us are just so shallow we'll not figure out how to live meaningful lives in an AI dominated future. The world is already meaningless. Some people live in the US, get tech jobs and live lives of almost indescribable luxury compared to someone born in South Sudan. There are beggars on the streets of most African countries, who are deformed of malnutrition and suffering in pain. There are children who will never know true happiness, and die in misery.
It's all fucking senseless and nasty, and as someone who originated from such a poor country it is infuriating to see spoiled developed world complaints sometimes.
We live, suffer, die. Some suffer less, some a lot more. We find out own meaning somehow, in this brief weird experience that we perceive as conscious life.
If an AI can bring a few billion 3rd world people out of misery, I don't think I'm going to shed tears if that means you have to somehow "find meaning" outside of a high paying tech job.
The worry “how will people find meaning with super-abundance?” has also me as tone deaf ennui of the already super-privileged. That we should be so lucky to have this worry.
> Some people live in the US, get tech jobs and live lives of almost indescribable luxury compared to someone born in South Sudan. There are beggars on the streets of most African countries, who are deformed of malnutrition and suffering in pain. There are children who will never know true happiness, and die in misery.
It's a bit baffling to me that folks think that the reason so many live in poverty now is a resource problem, and that an abundance of resources from AI would solve this problem. It is certainly a resource allocation problem, so I don't understand why folks believe that AI or our AI overlords will decide that everyone gets 40 acres and a mule.
I don't think most people care if it does bring a few billion people out of poverty, that's the point. They don't care either way, they just want their "meaning" (well paid US tech jobs) preserved.
> infuriating to see spoiled developed world complaints sometimes
Maybe glance back at your comment history and find you mostly are complaining about very developed world things and investigate why there might be a disconnect between these two aspects of you?
> If an AI can bring a few billion 3rd world people out of misery, I don't think I'm going to shed tears if that means you have to somehow "find meaning" outside of a high paying tech job.
Well, this is begging the question (how awful are people for worrying about the future when the future is going to be so great?), but you're also missing that there's more to humanity than surviving or not, and there always has been. There's no reason to reduce this to a dichotomy.
One related problem I see is the pipeline for producing working mathematicians seems to have been completely and irreversibly decimated. What’s the point of doing a long and arduous PhD when all PhD level research problems that used to take months to years can be solved by far less talented people with $100/$1000/$10,000 to spare? How do you even select people into your program (this part is likely hypothetical, classical talent selection probably still works at the moment, but what about in a couple years)?
Disclosure: I did a theoretical physics PhD, but got admitted to quite a few top math programs back when I was applying to math and physics programs simultaneously. If you asked me whether I’d do a PhD today I’d say why bother.
Here most local STEM PhDs try to get into finance. This is largely due to lack of funding for science and poor opportunities for PhDs. Why be a poorly paid postdoc when Jane Street is offering a million USD signup bonus ?
PhDs from poorer overseas do try to get related jobs here, mainly to be able to get a permanent resident visa.
> Why be a poorly paid postdoc when Jane Street is offering a million USD signup bonus?
One reason is despising that line of work. Quant firms were pummeling my @prestigious.edu inbox throughout my PhD and I fucking hated those parasites. Well, jokes on me if AI shatters my current career.
Btw a former classmate Caroline Ellison who went to Jane Street eventually landed in prison, lol.
Why put @prestigious.edu and name a famous person? Why not just say you went to Stanford? Lots and lots of people have gone to Stanford.
prestigious.edu isn’t Stanford; went to a different school for PhD. And the prison bit is just something funny about Jane Street (that sort of vindicates my view of these people) that I edited in later, didn’t think much about it.
Rephrasing the argument made in the post, math was like a "take a book, leave a book" exchange where you solved a problem, and in the process you discover more problems so the pool of interesting open problems is renewed. What OpenAI and the other labs are doing is akin to
- taking all the books in the exchange (which is technically allowed)
- destructively scanning them (still technically allowed)
- and not leaving any new books in their place (which is frowned upon, but there's no rule that says you have to replace books you take)
> but at the cost of sustaining the ecosystem
John Naur said the same of coding eons ago [1]. That code is only as healthy as the human mental model of its maintainers. Code is not the artifact, it is the intuitions and insight and mental models of its human maintainers. For a project to sustain, it is vital that the human mental models are kept alive.
[1] Programming as Theory Building https://pages.cs.wisc.edu/~remzi/Naur.pdf
How is this any different from people in any field that are impacted by AI and lose the utility of their skills and endeavors over the past decades? Are we saying that we're running out of problems to solve because of AI and hence it should be stopped? I am not underestimating the importance of the collective knowledge of the mathematics community and the role of mathematics as the enablers of other sciences, but opposing meaningful progress in that discipline or any for that matter feels counter intuitive. I would rather have the mathematics community start collaborating closely with the this newly evolving and powerful tool to expedite humanity's progress.
Do you see an end state in this? When AI is better than humans at everything (and I used to be very sceptical of that claim but I'm getting less and less by the day), I don't see the Wall-E version of humanity as some sort of utopia, and that's the good outcome.
To paraphrase the not-so-great philosopher Ted Kaczynski: Either we will maintain control of the machines or we won't. And if we do, it won't be you or I who control them, but a small group of elites.
His point is twofold: that the process of solving the problems leads to more than just solving the problem in front of you but other interesting things (he has an example of going on a hike to a waterfall and all the other things you might spot over in the distance or nearby on the way, which you’d miss if you were able to jump straight there), and also lots of the simpler open problems are ones early researchers learn on (this is akin to the “if we automate junior engineers how does anyone learn to be a senior?”).
Ability to jump straight to waterfall doesn't prevent you from taking another hike looking specifically for those other things.
If solving any problem is possible with ai, the job of mathematician is finding new problems, not trying to do what a machine can do better.
As for training, most of it is already done by solving known problems, and no one was suggesting to forget analytical geometry so that solving school geometry problems becomes cutting edge research.
Seems to be it, though Im not particularly concerned about this problem personally.
The fact that we all readily accept that modern AI systems can likely solve any math problem that no living genius can, tells me that no task is beyond this system we just need the right harness around it. The exhaustion of meaningful math problems to motivate mathematicians minds seems to be the least of my worries at that point.
Inb4 someone suggests that this is not proof that these AIs generalize, I agree thats a popular opinion, but both sides are merely that, with no possible way to prove. I will wallow in my existential dread while you do whatever it is that gives you comfort.
There was a recent link on HN about Tao explaining Six Essential Mathematical Concepts https://news.ycombinator.com/item?id=49503521
Here's an excerpt of the video where he explains his idea: https://www.youtube.com/watch?v=svl_1upFpQo
My summary: Science is like a hike. You have a general sense of what direction you want to go. You explore things along the way. You might make wrong turns but might also discover something new and interesting. While on your journey, you might spot new mountains or waterfalls or other vistas that look like they might be good to explore.
In contrast, AI is like taking a helicopter to your destination. Yes you got there quickly, but you missed a lot on the way.
Related, Tao later talks about the pipeline. It used to be that seminal proofs were rare, so there was lots of time for the community to review them, process them, share them, and summarize them in textbooks. Nowadays, AI helps a lot with generating more proofs, but not so much with the other parts of the pipeline.
Tao also explains the math pipeline issue in his talk at the International Congress of Mathematicians, also shared on HN last month: https://news.ycombinator.com/item?id=49056620
My overall take: Tao is very thoughtful and insightful about progress in AI and math, and about what the tradeoffs are. We're seeing similar problems in computer science (my field), where conferences are now getting over 10x the number of submitted papers over just a few years ago (not an exaggeration). It's straining the research community in ability to review, understand, and present the papers.
Indeed it's as if we've harvested all the large oak trees and can no longer make trans-Atlantic wooden ships and now must move on to metal. And doing that actually required stumbling on coal which ultimately was old wood. And then metal enabled us to move past coal. None of this was necessarily visible at the time but is the arc of progress. Of course mathematics being a purely abstract thought medium may not have the benefit of the physical / material world's progression. However it is ironic that here mathematics is progressing specifically because of how far along our material world has come. We're already at the point where a tiny nearly nil percentage of the population can reason at the frontier in mathematics, Terrence Tao obviously included in that small number, and if in the future that number truly reaches nil and only computers can go forward it's not so clear how fundamentally different the world will be to me. The hope is that the innovative manifestations in the real world will be able to be enjoyed by all just as one hopes mathematicians' thought exercises do unlock real blapplications.
I think the deeper point Terrence is making is simply that the information revolution which is the culmination of the mathematics revolution has less low-hanging fruit today. That was true ever since Claude Shannon
[dead]