Unless you're a really old fart, people were talking about AI safety long before you were born. AI safety issues do not go away depending on who gets funding. AI safety issues do not go away if the US or China makes the model. AI safety issues do not go away if it's an open or closed model. AI safety issue do not go away if the model is running at your home or at a data center. AI safety issues do not go away if $1 is being spent or $1 trillion dollars is being spent.
The fact there is no moat makes things far more dangerous. When LLMs start acting like weapons governments will treat them like weapons much to your dismay, crying, and gnashing of teeth as your door is kicked in and you're dragged out by armed men for running one.
Cast away your preconceptions for one moment and think "What will the future look like if LLMs are/can be actually dangerous".
I'd never claim AI can't be dangerous. It's a technology, and a powerful one, so of course it can. All technologies can be dangerous. The first early hominid to sharpen a stick probably spent some time regretting it. "Maybe we are not ready for sharp sticks..."
I'm specifically responding to the much stronger claims of people like Yudkowski and numerous other "AI safetyists" who assert very unlikely or in some cases impossible runaway superintelligence magical god machine scenarios.
I'd rank the risks as:
(1) Mass youth unemployment and an economy with zero entry level jobs, maybe not even that many mid level jobs, and even higher wealth inequality. This will lead to extreme political instability and populist revolts and revolutions, and unfortunately history shows that the new boss is worse than the old boss >50% of the time (at least, probably conservative).
(2) Mass propaganda, disinformation, and con artistry at scale, basically what we've already seen with mass manipulation through social media but supercharged. I think it's already happening but I also think we're in the early days.
(3) Automated mass surveillance and automated policing paired with authoritarianism, which could easily emerge from (1) and (2). Not only is Big Brother watching, but Big Brother is carefully evaluating everything he sees in real time and has a drone army to dispatch in seconds. The "prison planet" scenario (not the Alex Jones fever dream, but the realistic one).
What all those have in common is: the AI didn't do it. The AI (if it's ever sentient at all) is wholly innocent. All the likely nightmare scenarios I can imagine result from malicious or just short-sighted human agency amplified by AI.
Just like the sharp stick. The stick didn't do it. Someone stabbed someone with the stick.
The wild sci-fi superintelligent "Skynet" scenarios detract from discussion of those scenarios. Instead of discussing the very real, very plausible, already actually happening AI risks, we're talking about scenarios from soft sci-fi and Anime.
I wonder if a motive to heavily fund this type of "safetyism" is that all the scenarios above (the realistic ones) lead to discussions of things like wealth inequality, whether we should have universal health care or basic income, whether our system of taxation would have to change in fundamental ways, or what new limits should be placed on surveillance or policing in the AI age.
Those are discussions the very rich and the political class don't want to have. They'd rather us worry about Roko's Basilisk or magical runaway superintelligence inventing nanobots, wild bullshit that won't happen.
It's the same reason those powers would rather people argue about the culture war or inane conspiracy theories. I guess Yudkowskiism is the inane conspiracism of the AI safety discourse, a great distraction.