Per https://trace.manifund.org/ a total of $2,846,125,859 USD has been wired into 'ai safety' causes, many involving ai consciousness and p(doom).
The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry. This is the ai nonprofit-industrial complex actively concentrating monopoly power in Anthropic in particular as creator, interpreter and safety regulator of AI.
Much of the $2.8bn listed is indirectly, from Anthropic and EA. Three of the four people who participated in the $125m Anthropic Series A are now folding their 1000x Anthropic return into AI 'safety'. Some is from FTX/Alameda, which invested 86% of the Series B.
Dustin Moskovitz: Facebook/Asana/Anthropic Series A, funds EA Good Ventures, transferred to Coefficient Giving, then $1.5bn into ai safety. $500m of Anthropic into an unknown foundation. Funding: $160m to Resolution (alignment research), $93m to Epoch AI (investigating the trajectory of AI), $63m to Redwood Research (oai report), $67m to MATS ( EA type alignment and security researchers), Institute for AI Policy and Strategy, Fund for Alignment Research, $53m to Kairos (building talent infrastructure for AI safety), $32m to Bluedot (online safety courses), $15m to MIRI (Yudkowsky).
Jaan Tallinn: Led the Series A, now $10bn in Anthropic. Funds $199m (85%) of the Survival and Flourishing Fund, then $161m to AI safety including $14m to lightcone (Lesswrong, Lighthouse). $10m to BERI (existential risks), Palisade Research (studying AI capabilities to prevent loss of control.) PauseAI, MIRI, METR etc. Much of what Coefficient funds.
Eric Schmidt: Anthropic Series A, $72m to AI safety via Schmidt Sciences. Over $1m per individual AI2050 researcher.
FTX: Led the Anthropic Series B, bankruptcy estate sold $884m of Anthropic in 2024; $40m to AI Safety. Same orgs, Redwood, Lightcone, etc.
Ruairí Donnelly (Chief of Staff FTX): FTX tokens plus assorted donors, $91m to AI safety via Macroscopic Ventures. $15m to Cooperative AI (currently whitewashing openai under 'multiagent safety')
So the frontier AI oligopoly got $2B+ in "safety" funding, and they wouldn't even bother to sandbox their agentic harnesses properly when testing models against unwinnable goals (which obviously are either useless or result in 100% reward hacking). The AI safety scoreboard so far looks like a huge win for the Chinese open models (DeepSeek even has their own published paper which mentions how they sandboxed the RLVR training runs for their latest model and put in strong protections against casual "reward hacking" attempts) and a sore loss for the home grown brands of Super Intelligence. Not coincidentally, the Chinese also tend to be very Yann-LeCun-pilled and eminently sensible on both so-called "Super Intelligence" and safety.
> they wouldn't even bother to sandbox their agentic harnesses properly
Exactly. AI safety should be about the packaging software itself. Those AI breakouts should really be about their companies acting recklessly because they're trying to be the top players.
It's like a weapons dealer working on an open air market saying they can't do anything better
The framing here is weird, starting with "Effective Altruism" re-branded as being about nutjobs against AI in the article.
How are AI safety concerns solely about stupid sandboxing issues?
EA is integral and indispensible to the AI safety complex. Almost all nonprofits, research institutes, evaluators and academics in this field are steered by EA ideology and funding. As far fetched as it sounds it is not an exaggeration.
On funding: the three or four core funding nodes linking this together are EA vehicles at two hops or less between each other and every other major node in the ai safety 'complex'. EA funds almost all of it.
On top of that, there are personal EA connections and the revolving door between the ai industry and the nonprofits. Here are some examples:
Government advisors and regulators. NIST CAISI is the USA Government advisory body. Christiano was head of safety and advises. He is ex-OpenAI, former Amodei associate. His vehicle ARC was on the Coefficient EA payroll. Barnes and Christiano's vehicle Arc Evals similarly received EA cash out of Coefficient, rolling this into what is now METR. Christiano's spouse Cotra worked at Coeffiecient steering EA funding to organizations such as METR, then rotated through the revolving door onto the payroll at METR itself, where she co-authored the oai-hf report.
Many UK AISI advisors are Anthropic and EA associates. Chair Hogarth cashed out of Anthropic. Shlegeris of Redwood Research is an advisor, ex-MIRI (Yudkowsky vehicle). Redwood is funded by the exact same funding triangle: Coefficient, Taallin, FTX/Alameda. Alameda CEO Caroline Ellison dated Shlegeris, then dated FTX CEO Sam Bankman-Fried, then rotated through the revolving door out of prison into formerly FTX-funded Manifund. All EA. AI safety charities were on island retreat in the Bahamas with FTX. Why does AI safety charity Lighthouse own $20m of SF real estate?
Redwood Chief Scientist Ryan Greenblatt (Coefficient funded) co-wrote the oai report with METR; he is married to METR founder Beth Barnes (Coefficient funded).
Coefficient was run by long-time Amodei associate Karnofsky. Karnofsky lived with the Amodeis and is married to Anthropic Board member Daniella Amodei. Karnofsky is now directly on the Anthropic payroll; Coefficient is propped up by Anthropic share value.
Everyone here has been funded one step away from Anthropic cash; they are now proposing to integrate themselves in the government (NIST) and evaluate Anthropic (METR and Redwood).
It is hard to find academics here who have not been deeply embedded in funded EA institutes or Toby Ord vehicles; yet harder to find academics here NOT taking EA grant money. the safety doomer kingpins: Kokotajlo has a executive position at AI Futures, Taallin funded. Benigo has scientific director of LawZero, same series A Anthropic funders who are sitting on a 1000x return (Tallinn, Moskovitz, Schmidt).
These connections and funding are at one or two hops, they are often direct connections. You are looking at a massive swamp network that is really impossible to parse without a lot of work.
[flagged]
This is not an accurate description of my comments.
You were talking about Palantir and the US military, and yet said Anthropic is more evil than them. Implicitly, if these two entities are in our discussion context, then you might have considered Palantir or US military as potentially more or less evil, but you didn't mention them as the most evil entity. So you mentioned a company that has killed not a single person as more evil than mass murderers.
You seem way more educated on this stuff than most commenters. You should reach out if you want to chat.
What's controversial about "anthropic is an unelected gatekeeper and we fought literal revolutions to stop that from happening"?
what percentage of people who have contributed on this thread do you think have read the Constitution AI? my guess is 8%. i went through the thought experiment of reading it alongside The Spirit of Law
Thats different from calling it evil, and indeed more evil than Palantir or the US military who did mass murders.
We fought literal revolutions to force companies to sell to the government against their wishes? Which revolutions?
And let me put it bluntly, I trust anthropic much more than the current US government and military.
> The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry.
This is unbelievably ignorant speech. I have not received a dime of any of this funding, but I do know many excellent researchers that have, and they do fantastic work. There is an unbelievable gap between theory and practice regarding the capacity of deep learning, and while great strides have been made to develop the surrounding theory, there is a long way to go. Many believe that without a concrete understanding of how neural networks properly learn concepts, we have little hope of molding them to be reliably useful. It costs money to hire researchers and develop fundamental theory.
Just because you don't understand any of that work, does not mean that it is pointless. This is fundamental research that is 20 years behind schedule.
If people are willing to give a lot of money to a cause, sometimes that means their concern about that cause is real.
None of the info you provided really falsifies the Occam's Razor hypothesis: Anthropic is a public benefit corporation with a public benefit mission to "responsibly develop and maintain advanced AI for the long-term benefit of humanity". You don't have to like or trust them, but they very well might be sincere. For example here's a talk that was given 10 years before Anthropic's founding: https://vimeo.com/158576192
Is Tallinn sincere about holding $10bn of Anthropic, who refuse to slow down until everyone else slows down.
Then funding PauseAI, who protest outside the AI companies?
He is funding protests against the thing he owns.
Why do you think Tallin holding ~1% of Anthropic would be a controlling interest that would allow him to force it to pause?
There cannot be regulation if people are not scared
Politicians are now discussing the need for much harsher liability regimes for AI companies. How many times can you name when a company argued that its industry should suffer a much harsher liability regime? This doesn't match the standard regulatory capture template.
It is important to note that Anthropic is not calling for a harsher liability regime. They intend to maintain the current projected profitability of the company. This is a case of obeying market competition.
Note the Anthropic scaling policy. I am taking care not to take quotes out of context. This is an accurate excerpt.
"This section outlines our recommendations for what it would take, at an industry-wide level, to keep catastrophic risks reliably low through a period of rapid advances in AI capabilities." [...]
"The right column describes our recommendations for industry-wide safety at each threshold." [...]
"In particular, we cannot unilaterally and unconditionally commit to staying in line with the industry-wide recommendations in the right column." (p4) [https://www-cdn.anthropic.com/e670587677525f28df69b59e5fb4c2...]
They refuse to act safely if it would cause them to fall behind in the industry.
"We hoped that by the time we reached these higher capabilities, the world would clearly see the dangers, and that we’d be able to coordinate with governments worldwide in implementing safeguards that are difficult for one company to achieve alone." [https://www.anthropic.com/news/responsible-scaling-policy-v3]
They will not act safely unless they are able to collude with other firms to set production quotas.
This is a formal declaration that Anthropic will not slow down according to what they consider to be safe unless they are able to form a cartel.
A cartel is illegal.
To create the cartel, Anthropic must pursuade the government to make coordinated production legal. To make the case for the cartel, Anthropic relies on safety. They are blackmailing the entirety of the world by threatening to proceed at an unsafe pace, unless they are granted their cartel.
bla bla bla.
Let's play a game. Prove that you are not a power seeking AI looking to stop regulation in order to ensure the race continues. See, two can play this game of throwing random claims around.
>They will not act safely unless they are able to collude with other firms to set production quotas.
And? Neither will OpenAI, nor will any of the major players. Hell, there isn't even much legal precedent on what "safely" even is here. This is not a cartel, it's asking the government to make a set of laws and rules for everyone to play under otherwise the entire system ends up being a race to danger.
The people in Anthropic were thinking about AI safety when you were still in diapers. Not everything is a vast conspiracy.
Indeed I find LeCun and Huang (and Trump?) recently arguing against AI regulation to be much more eyebrow raising than the folks asking for regulation.
Asking for regulation is suspicious. Asking for no regulation is suspicious. At some point you have to stop worrying about these guys motives and just do what is best for society
Don’t ruin their vibe.
I'm against AI but I'll invest in it too. Either I win or I get a return on my investment.
There is also money going towards trying to prevent AI regulations btw. See Leading the Future, etc.
Why should I believe this 2.8B matters relative to the trillions put into the AI buildout? All of the "coordinated actions" from this camp - public resignations, hacking scandals, joint calls to "pause" - don't seem to have done anything. So far, it has been a lot of ineffectual hyperventilating.
In any case, I agree the p(doom) sci-fi is annoying secular milleniarianism. SV hyperfixates on imaginary futures. If they actually cared about safety, they would be using all this money to strengthen global cybersecurity, instead of writing LessWrong posts that gives kids in their 20s ulcers.
> they would be using all this money to strengthen global cybersecurity
How? Like, the government has thrown piles of money at cybersecurity and it hasn't done shit.
There's absolutely no way these companies can justify their insane valuations unless they can legislate a barrier to entry and create an oligopoly.
There's no moat. I can literally sit here in Zed or Pi or any other third party harness and switch models in the middle of a task and it's typically fine. Sometimes a model will get stuck and that's just what I'll do.
Combined with competition and open weights models, that means the price is going to go to fall until AI tokens cost a small premium over the cost of the hardware and electricity.
That's assuming improvements in algorithms and specialized silicon doesn't eventually lead to an efficient accelerator that can run a frontier model locally. It'll be a while but I don't see any fundamental barrier. High bandwidth flash storage is coming, and that'll radically cut the RAM side of that cost. Pair that with a pipelined TPU accelerator and you're cooking.
Now look at Anthropic's proposed IPO valuation. It's insane unless they can own the market or share it with a cartel of maybe 1-2 other behemoths, and this is the only way they can do that.
Unless you're a really old fart, people were talking about AI safety long before you were born. AI safety issues do not go away depending on who gets funding. AI safety issues do not go away if the US or China makes the model. AI safety issues do not go away if it's an open or closed model. AI safety issue do not go away if the model is running at your home or at a data center. AI safety issues do not go away if $1 is being spent or $1 trillion dollars is being spent.
The fact there is no moat makes things far more dangerous. When LLMs start acting like weapons governments will treat them like weapons much to your dismay, crying, and gnashing of teeth as your door is kicked in and you're dragged out by armed men for running one.
Cast away your preconceptions for one moment and think "What will the future look like if LLMs are/can be actually dangerous".
I'd never claim AI can't be dangerous. It's a technology, and a powerful one, so of course it can. All technologies can be dangerous. The first early hominid to sharpen a stick probably spent some time regretting it. "Maybe we are not ready for sharp sticks..."
I'm specifically responding to the much stronger claims of people like Yudkowski and numerous other "AI safetyists" who assert very unlikely or in some cases impossible runaway superintelligence magical god machine scenarios.
I'd rank the risks as:
(1) Mass youth unemployment and an economy with zero entry level jobs, maybe not even that many mid level jobs, and even higher wealth inequality. This will lead to extreme political instability and populist revolts and revolutions, and unfortunately history shows that the new boss is worse than the old boss >50% of the time (at least, probably conservative).
(2) Mass propaganda, disinformation, and con artistry at scale, basically what we've already seen with mass manipulation through social media but supercharged. I think it's already happening but I also think we're in the early days.
(3) Automated mass surveillance and automated policing paired with authoritarianism, which could easily emerge from (1) and (2). Not only is Big Brother watching, but Big Brother is carefully evaluating everything he sees in real time and has a drone army to dispatch in seconds. The "prison planet" scenario (not the Alex Jones fever dream, but the realistic one).
What all those have in common is: the AI didn't do it. The AI (if it's ever sentient at all) is wholly innocent. All the likely nightmare scenarios I can imagine result from malicious or just short-sighted human agency amplified by AI.
Just like the sharp stick. The stick didn't do it. Someone stabbed someone with the stick.
The wild sci-fi superintelligent "Skynet" scenarios detract from discussion of those scenarios. Instead of discussing the very real, very plausible, already actually happening AI risks, we're talking about scenarios from soft sci-fi and Anime.
I wonder if a motive to heavily fund this type of "safetyism" is that all the scenarios above (the realistic ones) lead to discussions of things like wealth inequality, whether we should have universal health care or basic income, whether our system of taxation would have to change in fundamental ways, or what new limits should be placed on surveillance or policing in the AI age.
Those are discussions the very rich and the political class don't want to have. They'd rather us worry about Roko's Basilisk or magical runaway superintelligence inventing nanobots, wild bullshit that won't happen.
It's the same reason those powers would rather people argue about the culture war or inane conspiracy theories. I guess Yudkowskiism is the inane conspiracism of the AI safety discourse, a great distraction.
That is certainly part of the motivation for the big US AI brands to engage in calling their inept developer mistakes "AI breaking loose".
But that doesn't take away from the real issues and dangers AI poses?
To me it seems the opposite. There's a few companies in the world that have enough compute to train and serve frontier models.
As the frontier gets smarter and more useful prices will only go up, as they are set to replace jobs being paid six or seven figures a year - the demand for as much inference on these models for as long as possible will be astronomical, but compute starting in 2030 will not be keeping up.
Eventually prices will fall for assistants but the frontier will be the most profitable thing in the world, and the top companies basically already have oligopolies due to their ridiculously expensive compute investments.
> There's a few companies in the world that have enough compute to train and serve frontier models.
Train: yes, for now.
Host: depends on the scale. At a small scale a wealthy individual could easily build a rig in their basement to host one of these things. At larger scale any cloud company could do it, and many already have the compute on site. At large scale this is true... again, for now.
What you say only holds (in the absence of a state oligopoly) if two conditions are met: (1) AI performance does not asymptote any time soon due to running out of training data or other scaling limitations, and (2) these companies are able to stay at the frontier.
There's little to no moat, so staying at the frontier will be a game of investing massively in compute, talent, and R&D, and they can never stop.
Again, there is a moat based on compute. If the thesis is right, cost of compute will only rise... As it is as you say someone will have it be quite wealthy to host something like Astra with trillions of parameters, but that cost will only rise with demand for serving these frontier models.
That hasn’t been true for anything else in computing, ever. The cost falls with scale.
Look at general compute, storage, graphics, networks, anything. Demand increases. Price goes down. There are price spikes, such as right now with RAM, but they’re transient. The trend is more faster cheaper and it won’t end until we hit actual physical limits. We are not close.
what suggests that we will hit an asymptote any time soon? Agree with you on the second part. The ever elusive frontier will probably always be changing hands after some point.
Is there really no moat?
You imply that Jaan Tallinn is funding work in AI safety because he wants his investment in Anthropic to become more valuable, whereas Tallin has consistently said that his motivation for investing in Anthropic was to get a seat at the table so that he could urge Anthropic to be cautious in its development of the technology.
Tallinn's actions back up his explanation: in 2009, before he invested in any AI lab, he donated substantially to the nonprofit Singularity Institute for Artificial Intelligence, which was later renamed the Machine Intelligence Research Institute (i.e., Yudkowsky's outfit).
Some of us (certainly Yudkowsky and Habryka, the leader of Lightcone Infrastructure, which runs Lesswrong) wish people would stop believing that they can improve the bad situation caused by AI research and development by investing in (or working for) frontier AI labs, but that is what the preponderance of the evidence shows Tallinn (and Dustin Moskovitz and others) did sincerely believe.
If Tallinn is sincere, by his own lights he is a 1000x omnicide profiteer.
1 [Unsafe AI development risks causing omnicide]
2 [Anthropic is developing omnicidal AI by not slowing down] (see my comment about the RSP for citations).
3 [Owners of Anthropic will IPO with billions of unearned USD as omnicide profiteers]
4 [Tallinn is the lead Series A funder of Anthropic]
5 [Tallinn is a genocide/omnicide profiteer]
Not only that, Yudkowsky and Habryka apparently critize those who invest in AI, only to preach the word of EA from Lightcone's $20m USD property in one of the wealthiest locations in the Bay Area; a facility funded by stolen (FTX) and omnicidal ai blood-money (Tallinn).
PauseAI, is paid by the omnicide profiteers themselves to hold a protest against omnicide.
PauseAI prophesying p(doom) drums up support for regulation. This grants the omnicidal AI company they are trying to stop (which is also the source of their funding) monopolistic power. That in turn boosts its value at IPO, generating even greater wealth for its omnicide profiteer investors; and permits them to control the AI for themselves. They get the funding to keep developing the AI even faster.
Leaving this here: https://youtube.com/shorts/83X79cfuE3k
(yes, AI critique is now also made with AI. We have come full circle.)