A few predictions:
1. I fear that AI is about to change from being a human art to becoming something like a bulk-extruded, industrial product.
2. I also suspect that mathematics is about to become substantially less open. As the number of entities on the planet that are capable of top-level maths explodes, and because most of those entiries will have no interest in the validation that publishing papers brings, we will see balkanisation and hoarding of "secret maths".
3. Areas of science that are downstream of maths -- basically everything -- will, in time, be degrated by lack of openness, before suffering the same fate.
I'm an academic mathematician, and could you explain your second prediction?
It goes entirely against the culture of research mathematics to hoard secrets. If you've done something interesting, you want to share it, and it's in your best interest to share it.
Of course, this culture could change under external pressure from AI tools. But, aside from cryptography and the applied fields, I'm finding it difficult to imagine any possible motivation for spending substantial effort on top-level math and then keeping it secret.
I can see it happening if people start depending AI-generated proof libraries for the work they are interested in. Not because the libraries are proprietary, but because they are impenetrable. AI usage wouldn't result in this directly, but it could be a consequence of formal verification as a substitute for canonicalization in some cases.
(Disclaimer: I had quite a bit of formal training in mathematics at one point, but have worked in software for more than two decades now. I wasn't on the applied side, but I recall that MATLAB evoked similar concerns back in the day.)
I'm not a mathematician or any other kind of scientist. Just a toiler in the software mines, and an onlooker.
> It goes entirely against the culture of research mathematics to hoard secrets. If you've done something interesting, you want to share it, and it's in your best interest to share it.
You're describing an enviable human culture with a reward system suited to people. People publish papers etc. to enrich the human domain and to get recognition. AIs don't care about things - and neither, I'm asserting, will their owners. That culture will become a vestige.
Mathematics as the underpinning of science and technology that can be exploited for inter-state competition, including cryptography but not limited to that (image analysis, targeting algorithms, conflict modelling, whatever) won't see the light of day. The AIs that discover the mathematics will be paid for by the warfighting budgets of states, and by the corporations that feed on them.
There could be a difference between academic mathematical research and commercial mathematical research, just as in the difference between FOSS and commercial code. Perhaps the difference lie between pure math and applied math with applied math having more incentive for secrecy given that large swaths of computing is just applied math in fancy wrappers.
If you can derive some special formula that prints money from accurate prediction of asset valuations if and only if you're the only one with the efficiency, you're incentivized to keep it quiet.
If you're a nation state that's cracked a simple method to decrypt modern "pre-quantum" encryption schemes with less compute/wall time using a development in mathematics, it is in your interest to keep this quiet so that the parties you surveil to not switch or adapt. Similarly if you've figured out a handy new form of encryption, you're incentivized to prevent leaking of details as it is harder to reverse such operations without details of the methodology.
Or you've designed some special optimizer that you then bundle in a subscription service. Your business now only profits while the algorithm is secret. e.g. The DVD Copy Control Association's CSS encryption was only a valuable thing that the company could charge for or entice clients with before DeCSS was published/became popular.
There are different levels of outcomes, but even Terence Tao essentially foresees this issue: look at his remark footnote 5 in this very talk. It complements the Balkanization point in the parent comment. Not a big leap or unfathomable to connect to broader issues.
Update: see also the later slide where he problematizes corporate secrets in relation to math culture.
Everyone should just read the slides, really
Wouldn't AI make validating the research much cheaper?
Well, that's what AI does. It makes everything about the final product and nothing about the enjoyment of the process. Now a lot of people here will say "you can still read the AI results and enjoy math on your own time", but that is completely at odds with how most people do things: a large part of most people's enjoyment comes from sharing results with others and AI devalues that. People can't help but feel less impressed when you can come up with a result but AI can also.
People on here have also argued with me that enjoyment and fun should be decoupled from work but I'd argue that jobs that are enjoyable are important because I mean, what really counts in learning is inspiration. If no one is having fun, no one will be inspired to learn and do math in the first place. The best professors I've had are clearly having fun.
And sharing that fun on a human level is exceptionally important.
I just don't see the tradeoff (more math at the expense of people being able to have fun expressing themselves), as being worth it. The whole point of discovering knowledge is to enjoy the process and share that with the world to create hope an optimisim in it, not to product like a factory.
Unfortunately, this website tends to bring in the rare type of person who loves to believe that they can enjoy things all by themselves, and maybe they can, but the problem is that they (not necessarily you) foist their unusual view of the world on the rest of the world by developing and supporting AI and get defensive when someone points out that it actually contributes to a horribly destructive social structure - a cognitive dissonance most of them can't handle.
> The whole point of discovering knowledge is to enjoy the process and share that with the world to create hope an optimisim in it, not to product like a factory.
This is the crux of everything you've written and I strongly disagree. The purpose of knowledge is the sheer power it brings, and if there's factory scale churning out of it, I'll gladly take it.
As a closer analogy, consider this probability (I'll be stealing from the game Stellaris here): let's say a few hundred years later, when we are spacefaring, we discover a vast cache/library of knowledge of an alien civilization, we won't just pass up on exploiting it because we need to "enjoy the process of acquiring it on our own" or "derive the satisfaction of sharing the discovery with others". No, I and I daresay most others will take as much as we can without restraint in a quest to make all our lives better/more enjoyable.
And if that's ok to do, using machine intelligence to mine/grab any new knowledge or new math as Tao is suggesting here should also be acceptable and welcomed.
> The purpose of knowledge is the sheer power it brings
A dismal prospect.
When knowledge is acquired at a human scale, and under the influence of a human culture, then extraordinary things can happen. Agriculture, the transistor, Gödel's theorems.
When knowledge is "churned out" at "factory scale" then human flourishing isn't even a by-product.
> This is the crux of everything you've written and I strongly disagree. The purpose of knowledge is the sheer power it brings, and if there's factory scale churning out of it, I'll gladly take it.
Actually, this is the fundamental divide. There are a group of people (like you) who want the world to be one way, and there are a group of people (like me) who think that world is abhorrent. I don't think we should go after that power.
But, I'm glad we got to this point in the discussion because it shows this irreconcilable divide. The truth that most people will never say is that these two groups are enemies and will forever be. They fit into this world because we currently participate in the same global system, but because of this situation, it's an unstable one. And AI is exactly the system that is distilling this antagonism to its limit.