I have never understood this line of thinking. Why would a company warn that their technology is dangerous if it wasn't? Looking at the ongoing Mythos export control issue, it's obviously not good for profits. If you assume AGI is possible, you must agree that it would be at the very least profoundly destabilizing. The companies are built around this assumption.
> Why would a company warn that their technology is dangerous if it wasn't?
Because it makes their technology seem more important and powerful?
Do you usually believe 100% what a company says about itself?
> Mythos export control issue, it's obviously not good for profits
Who cares about profits? None of these companies make money like a business currently. It's all speculative valuation and influenced by the same hype/doom cycle
When people discuss the negative effects of climate change, do you think that boosts fossil fuel company valuations by making their products seem more important and powerful?
Hell yes. More than a third of Americans would run their two and a half ton pickup trucks on burning copies of An Inconvenient Truth if it were at all possible
Fossil fuel companies do everything in their power to stop that discussion.
However, arms manufacturers are doing everything in their power to make you feel unsafe and weak and scared. That way, you are more likely to buy a gun.
> Looking at the ongoing Mythos export control issue, it's obviously not good for profits.
This is not obvious at all. Anthropic's biggest investor was the catalyst for this action in referring the "jailbreak" (if you can even call it that) to the government. Now, Anthropic is sitting around a table with government officials who are designing benchmarks that will determine what Anthropic's competitors will be allowed to build. At this point, I have no evidence to rule out the possibility that Anthropic's leadership purposefully sought this outcome.
*why would they warn if it they didn’t believe it.
2 words: regulatory capture.
Ban Chinese models because they’re “bad”. (can’t have head to head competition).
Bad open weight model (can’t let people escape the subscription model).
People have been warning about the dangers of advanced AI since long before open weight competition was a consideration.
Sure, but they’ve either been the kool-aid-drinking p-doomers, or academic discussions about giving them access to weapons.
Now we have real companies whose valuation depends on people using their services.
yes, and all their abstract, philosophical thought is now being used to justify high valuations by these companies. if what they are making is the new nuclear bomb, well then they must be very valuable indeed.
> 2 words: regulatory capture.
> Ban Chinese models because they’re “bad”. (can’t have head to head competition).
The current scale of current + planned data centres only makes sense if the US exports inference worldwide. 30% of your current total electrical generation capacity, and equivalent to 76% of your total electrical demand in 2022. At least, assuming the "33 GW installed, 300 GW more planned" claim I've heard was not itself an AI hallucination.
And at least OpenAI and Anthropic were persuing the same argumemts continuously to their initial founding, before such capture was plausible.
I don't only mean that GPT-2 was, famously, when OpenAI surprised the world with (paraphrased) "we think this is a good point to start practicing not releasing the weights, before models get too dangerous" or as a surprising number of people interpret it even when I link to OpenAI's own post "we think this is … too dangerous".
I mean even before that.
I wouldn't disagree if you say Musk is full of BS, but he did donate to OpenAI when it was created as a nonprofit, and his wealth was 2 orders of magnitude smaller than today. Back then he was talking about risks, "summoning the demon", now he's the one demanding saying "robot army" to justify a bigger compensation package from Tesla.
>> Looking at the ongoing Mythos export control issue, it's obviously not good for profits
We must live in different planets. The whole episode about Mythos has been the absolute best free PR move ever designed. At this point everyone in tech assumes Anthropic and Mythos are the absolute best models. All of this got achieved without spending a single $$ in advertisement.
This model is also probably really dangerous, not denying that. But they definitely used it as a huge PR tool.
> The whole episode about Mythos has been the absolute best free PR move ever designed.
Great PR.
For a product nobody can pay for.
And a high chance that when anyone can, the market shrinks from "worldwide internet users" to "the USA only".
And it's known to be a highly dynamic market: based on the historical average, it only takes 11 weeks to be completely trounced by someone else's model, so if you're out of the market that long you may as well not have bothered.
How many non-tech people knew about Anthropic before? How many know about Anthropic since this episode?
Still don't see it? Still don't see how the huge IPO coming up soon could profit from this?
Yes, this was a brilliant PR move (independent from the actual benefit of the model).
> Still don't see it? Still don't see how the huge IPO coming up soon could profit from this?
"Would you like to buy Anthropic shares?"
"Anthropic? Didn't the government ban them?"
"Yes"
"I'll pass"
>For a product nobody can pay for
I remember then Thefacebook.com was only available at certain universities and everyone felt slight envy that they weren't included. Didn't work out too bad for them.
Do you remember them being hit with an export ban that also included their own staff?
You could also turn this around: Why would a company put up warnings about their product, or at least more than they're legally obligated to. The tobacco and social media industries certainly didn't go "this product has the risk to do great harm, but we also believe in its potential, so we'll go with a responsible approach".
The only examples of companies voluntarily putting up warnings I can think of are those obviously tongue-in-cheek warnings "this book/game/series may cause addiction because it's just so good" etc.
When the U.S. first legalized pharmaceutical ads on TV, one of the requirements was that the drug companies had to disclose side effects during the ads. The companies chafed at that, thinking it would make the drugs less appealing, and only included the bare minimum the rules required them to. But their own data eventually showed that consumers actually responded positively to hearing the side effects. They found that people think if a drug is powerful, it must also have strong side effects. So instead of just scaring them, it reassured them the product was good. After learning that, the drug makers started including more side effects in the ads than they were required to.
I agree. There are 2 groups of people:
1. Those, like OP, that believe AI danger and disruption is all hype to boost valuations.
2. Those, like myself, that believe AI danger and disruption is very real and presents dilemmas (but also opportunities) for society, so we must tread carefully.
The first position is not logically consistent with reality. The danger is real: we have already seen hints about how this will impact everything from jobs to warfare to mental health. that danger does not increase valuation, in fact it does the opposite because of the need for government regulation.
> in fact it does the opposite because of the need for government regulation.
For frontier leaders, regulation is a competitive moat. At trillion dollar scale, compliance teams, legal and lobbyists are a tiny fraction of opex. For emerging startups, early regulation of a new market is a substantial barrier to entry.
That said, I think Anthropic blundered into their current Fable mess unintentionally by underestimating just how much they pissed off the current administration by refusing to let the DoD use Ant's AI's for all the domestic spying and lethal combat ops they want.
I don't think the first position is steel manned by assuming those holding it actually believe there are no potential problems or ways to use the technology badly.
But that belief does not imply the doom rhetoric is necessary or the best way to approach those issues.
[dead]
that's an inaccurate characterization on #1. he's not saying ai risk is _all_ hype to boost valuations, but that it's misused to serve purposes other than warning.
Those words come directly from TFA:
> It’s all just nonsensical hype
And he’s referring to completely real and reasonable warnings in anthropic blog posts about the exponential rate of AI development.
What's "exponential" about AI development? Model parameter counts? Anthropic doesn't publish those for their own models, last I checked. Datacenter buildouts? Water consumption per request? There just isn't enough evidence that AI smarts is growing all that much, once you account for the scaling of inputs. That's what OP probably means by "nonsensical hype".
What's "exponential" about AI development?
The METR task-completion time horizons, for one.
https://metr.org/time-horizons/
Lousy benchmark, they explicitly focus on the easiest tasks to automate for AI (i.e. heavily cherry picked outcomes) and it seems that they don't bother to test anything except just-released proprietary models.
> Lousy benchmark
Make your own then. It can go on the pile with all the others that keep getting saturated too fast to be useful.
> they explicitly focus on the easiest tasks to automate for AI (i.e. heavily cherry picked outcomes) and it seems that they don't bother to test anything except just-released proprietary models.
What?
They made the benchmark last year, and included a bunch of models going back as far as 2019.
When they first announced it, the top end of their tests were things AI could not actually automate, and even now only does erratically. Examples of the tasks SOTA models are now saturating (at the 50% success level, not at 80%) include:
They're benchmarking against the time it takes humans to do the same things, which means everything they ask every AI to do must have also been done by a human.It's a blog. He's using hyperbole. It's not a Supreme Court opinion.
what the "tfa" doesn't say is that geohot writes like how most people talk. you're framing the observation you don't like around a single word and ignoring everything else.
Because it makes them feel important because they are developing something so dangerous everyone else ought to listen to what they say.
It's about ego, not profits.
Why would a company warn that their technology is dangerous if it wasn't?
The whole point of the article is to answer this question, and here's the answer:
Because all the AI doom fear-mongering is driving sky-high valuations. The more the public panics, the more investors open their wallets.
Is there any evidence of this? It seems like a thing people say all the time, but completely unsubstantiated.
What else is there? The product as delivered today doesn't come anywhere near a justification for these valuations. All of it is built on an expectation of future capabilities, and that's where the Dario-penned doom papers come in.
You cold also drive valuations like Musk does.
Hype up the *positives*.
Even here on HN, Tesla fans try to justify the market cap by pointing at Optimus, even though nobody can buy it and there's already competition for it which you can buy.
The positives aren’t there though, beyond what people are already using it for (pair programming aid, fancy auto complete, refactoring tool). Hence the article’s thesis: the doom justifies the valuation.
Lack of physical reality doesn't stop Musk from hyping in an endless loop. Or hyperlooping, if you will.
SpaceX announced a private flight around the moon in 2017 to launch in 2018, but instead in February 2018 SpaceX cancelled the certification of vehicle it was supposed to use and said they were switching to BFR. In September 2018, that became dearMoon and was due to fly there in 2023. Starship did actually launch in 2023, but exploded twice, and dearMoon was cancelled by the customer in 2024. Starship still isn't getting its test orbits circularised, because SpaceX doesn't trust the engines will relight correctly. What's in the IPO docs, to justify the price? Space based data centres that only make any sense (and even then barely) with cheap rockets that don't work right yet.
Can say the same for self-driving, or Optimus, or TBC, or Neuralink, or Cybertruck, or X, or Grok (where the closest he gets to "doom" these days seems to be how he considers "woke" a dirty word). And in the past even how many cars Tesla expected to sell vs. what they actually sold, which is a demonstration of how shifting goalposts, and not just missing them, still fails to undermine hype.
It's all "jam tomorrow" with him, even though when tomorrow comes there's still no jam; and yet, how many trillion in market cap?
[dead]
In fairness, it is true that there is a correlation such that:
> The more the public panics, the more investors open their wallets.
I don't believe this to be causation, but I can see the correlation.
>Why would a company warn that their technology is dangerous if it wasn't?
Because Americans are obsessed with eschatological narratives and action hero stories. I talked to a Palantir guy once who told me that he loves it when journalists describe his company as a James Bond villain because every single time the market cap goes up.
If they said that they're SAP with Call of Duty marketing they'd actually lose money. In other countries this strategy might not check out but in the US it's better to pretend to be a supervillain than to be boring
> If they said that they're SAP with Call of Duty marketing they'd actually lose money. In other countries this strategy might not check out but in the US it's better to pretend to be a supervillain than to be boring
Well, that explains Musk.
> Why would a company warn that their technology is dangerous if it wasn't?
That's a bit like asking, "why would someone claim to have a lot of money if they didn't?" It's step 1 in countless scams.
You can't think of any ways to exploit people by convincing them you have technology which might rule the world soon?
To me, they're selling the "power" of their product by mentioning the danger. "It's TOO powerful to even release yet!"
Whether or not that level of power exists, that is definitely how they're pitching it.
They banned exporting bad cryptography algorithms/code 30 years ago too.
People just love to talk about anything that tickling their fancies.
It’s all a fugazi.
Setting up for the eventual bailout. Remember - "we can't let banks fail or it'll be a depression"?
I remember a meme that spread to that effect, but it was similarly divorced from reality. Hundreds of banks failed during the financial crisis, including multiple which were very large. There were a few cases like Wachovia, where an unhealthy bank was required to sell itself in order to avoid a technical failure that would have impacted over 10 million Americans' access to their money. But this was still unpleasant for existing Wachovia shareholders.
I remember large banks that caused the failure to be bailed out. And talk of trickle down economics, except that average joe was not bailed outed, but punished.
I remember robo signing of home foreclosures with no checks and people loosing houses despite paying debts.
> I remember large banks that caused the failure to be bailed out. And talk of trickle down economics, except that average joe was not bailed outed, but punished.
Another thing that is commonly remembered but did not really happen as described. If you had to identify any particular bank as having caused the crisis, it would have to be Lehman Brothers, which was not bailed out. But really, it was a systemic crisis to which TARP was deployed as a systemic solution. Much of the "bailout" talk is based on memories of AIG, which is not a bank and behaved far more irresponsibly than any bank did; they seem not to have bothered hedging their exposure to widespread mortgage defaults at all.
> I remember robo signing of home foreclosures with no checks and people loosing houses despite paying debts.
This did happen, yes, and that was bad.
A common line of logic would be that a company would never put a dangerous sticker on their product unless they absolutely have to by regulation. So if they do it, it absolutely means they see a benefit of doing it.
Also why do you think the Mythos thing is bad for profits? Anthropic hyped the model up too hard and didn't even have the compute to serve it properly to all users.
The export control issue has not affected Anthropic's revenue and only made its top of the line products seem far more impressive than they actually are. People don't think Nvidia is a bad company if the government puts export controls on their products.
For Anthropic, valuation and public perception is 100x more important before the IPO than revenue or profitablity.
It’s the same narrative as “the communists are developing nukes so we need to develop first”.
the headline literally explains that it is to maximize hypothetical valuation
> it's obviously not good for profits.
Weird to think profits matter half as much as ever-increasing valuations, driven by memetic bullshit. E.g. the entire point of the article you're commenting on.
Anthropic's valuation is driven by its revenue growth. Export controls harm this. Hype certainly plays a role, but it's simply not the case that revenue is not a major part of the picture.
What revenue growth? They go from net loss in year 1 to larger net loss in year 2, always have.
Regulatory capture and false AGI hyping, what else? You'd have to be willingly ignoring the writings on the wall if you still haven't identified it yet.
They did not wanted export controls, but they did wanted everyone so scared and thus pay for license. The decision backfiring does not imply it was not intended to be self serving.
If the company really think their product is dangerous, they should stop making it increasingly dangerous. Antromorphic claims is dangerous and that they need more money to make it even more dangerous.
And the fearmongering is beneficial for them so far.
> Why would a company warn that their technology is dangerous if it wasn't?
It helps to look at the history of the AI Alarmist, EA-adjacent mindset. They believe AI could be Dangerous (<--- capital D) if not responsibly managed by the right people (spoiler: themselves or politicians who listen to them). The capital D means more than just some economic and employment disruption, they're thinking much bigger, like "existential threat to humanity" big.
The most extreme alarmists are basically LARPing Terminator 2 and want to shut it down before SkyNet takes over but the more moderate alarmists believe AI is too potentially beneficial to walk away from. And they acknowledge other nations and various bad actors won't stop anyway, so they believe it's their sacred duty to move forward quickly albeit very responsibly. Yes, this leads to some conflicted reasoning and priorities, like they need to keep gaining access to gobs of capital to ensure they are the ones to reach AGI first, as that's the only way to ensure "god" is benevolent.
While I'm sure many of the top AI leadership are primarily (or purely) mercenary, I also think some, if not many, other execs and employees sincerely believe (or at least hope) they are "the good guys" playing a crucial role in an era of historical importance. Obviously, this has some heavy vibes as well as a being quite a buff to one's self-importance. They face each new day feeling the weight of bearing such a consequential role in shaping the future of humanity. Us mere mortals can only pray their wisdom and altruistic ethics match their humility. </s>
So... yeah, that's why we see some weirdly whiplash messaging.