An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).
The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.
China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply.
But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer.
I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.
Yeah, more global competition in DRAM would be great.
SK Hynix and Samsung are South Korean.
> SK Hynix and Samsung are South Korean.
The Korean memory makers are playing the same game as Micron and simply moving existing capacity up-market.
GP was referring to upstart Chinese memory manufacturers like ChangXin, who - if their yields manage to catch the wave - could not have asked for a more favorable market after the big 3 have abandoned the consumer segment. Consumers who would have otherwise turned up their noses at CXMT will not have the luxury.
Chinese manufacturers will probably takeover consumer ram that most of us use as current manufacturing contracts expire and Samsung SK and micron move all their production to HBM for data centers. Corsair recently released chinese chips based DDR5 sticks.
I suspect Chinese factories will get built first, but quality may take a few years to really nail down.
Basically:
China floods the market with cheaper but less QA'd parts, makes a gazillion dollars, is able to spend said money to fix yields / QA issues and streamline operations, by the time that happens Micron and maybe a few other existing players will have new memory production, and then we'll have a flood of cheap, reliable memory. 4yr, maybe?
They're doing decent enough already for consumer electronics. Corsair is selling 16GB 6000MT/s CL36 DDR5 sticks in China using memory from CXMT: https://www.tomshardware.com/pc-components/ddr5/chinese-memo...
How long would it take an aggressive company to expand production capacity? I always thought it takes a few years, at minimum, for even established players to stand up new fabs
It is not a law of nature that Chinese products are lower quality (cf. electric cars) and I don't see why they would go for that. They can just bin what they produce like everyone else and sell their products for what they have been tested to deliver.
But it is a near law that the first to market attempts will fully embraces the deeply engrained culture of 差不多, until market forces beat it out of the product line.
This has nothing to do about nationality, it has everything to do with building and running a brand new, highly technical, mass production facility.
And historical record of the lack of QA coming from Chinese manufacturing
China wants a sovereign DRAM capacity. They're playing an entirely different game then the commercial suppliers in the West.
This is wrong. It is NOT in their nature to keep the market under-supplied -- eg, Samsung, the industry's largest company, was notorious for expanding their capacity during the industry downturn to gain market share while everyone else was cutting back to minimize loss.
I'm guessing you are also probably unfamiliar with the terms like "chicken game" which refers to the cutthroat, high-stakes price wars where dominant semiconductor manufacturers intentionally overproduce and slash prices. This is literally how the industry went from dozens to just three majors today since the 80's.
You're making the point for him. Undersupply in a boom, store cash to ramp up capacity in a downturn. Presevres capital and avoids overcapacity during the turning
Sure, but the key word here is "was"
The industry is so naturally prone to oversupply that the only stable equilibrium is undersupply. Aggressive expansion kicks off a price war, which immediately undercuts the logic of the expansion.
This only changes with new entrants, which will come, especially from China. But it takes time to build fab capacity, so the medium-term modal outcome is consistent undersupply.
If the existing memory makers retains control of the market and don't defect from the optimal-long-term equilibrium for themselves, that's true. It just takes one player to defect for short term gains as we've seen with some past boom-and-bust cycles. Alternatively, it takes a sufficiently-resourced player with enough incentive to enter the market themselves (NVidia, Google, Amazon, the PRC government through one of many companies...)
Relevant article posted on HN about this a few days ago: https://davidoks.blog/p/ai-is-killing-the-cheap-smartphone
Reminds me of how Samsung is giving out $340,000 per person bonuses. Shows you how much of a stronghold they have in market.
CXMT is scaling up incredibly fast, they are on a clock (south koreans) their monopoly will end relatively soon, although I'm guessing that the AI companies will crash before that anyways.
Supply and demand always balance out. There is no way manufacturers aren’t going to compete away these inflated margins, as long as they feel like this demand is sustainable.
You know there's other strategies? Companies can be more clever than naively undercutting each other...
Memory in particular ... https://en.wikipedia.org/wiki/DRAM_price_fixing_scandal
The entry-cost to getting into memory is on the order of $billions and years - you can do just about anything...
There's very few manufacturers, I believe 3 globally? And there's a large moat. Nobody can compete with them in the next 10 years. It's really not hard to coordinate action between 3 companies.
There are trillions to be made. That moat won't be as insurmountable in hindsight.
There used to be over 50 memory manufactures in the US alone. Everytime there was a bust (following a boom) there'd be bankruptcies. The lucky ones got bought out and consolidated. Empirically, attempting to capitalize on memory booms is a losing strategy.
There really aren't though. The reason there's only three is because memory is a commodity and margins are historically very low. It's not a very good business to be in, generally.
In the past when memory supply was short and then rebounded, many companies went out of business because making memory was no longer profitable.
Only in the most naive sense.
If it costs you $1B and five years to build out new supply and you think demand will not sustain for more than three years, it does not make sense to expand supply.
Instead you will maintain your margins currently and await demand to decrease back to your current supply.
This is pretty common and as others have pointed out is even more common in markets where competition is slow and lead times are long.
Ammunition is a great example over the last decade or so as political turnover caused relatively short lived demand spikes and manufacturers didn't expand supply because they knew once political winds shift, demand would decrease.
...which is presumably why GP said "as long as they feel like this demand is sustainable."
Apple could always decide to build their own fab or some such thing.
That’s not the Apple way, but they might fund a supplier to build out capacity in return for priority access.
The thing is they tend to only do that when they can get a technological competitive advantage. The priority access gives them a locked in competitive edge, for a while. It’s not clear there is an opportunity like that in memory.
It sure looks like Sam Altman's masterful gambit to corner the memory market has had unforeseen consequences.
Is any of this actually unforeseen? Buying the vast majority of the world’s supply of something does have mostly predictable consequences.
What’s the lifespan/refurbishability of the capex elements like the “GPU” modules or even the DRAM soldered into them?
For lifespan, AWS is still running a ton of T4 GPUs from 2018, that power a lot of computer vision models. A ton of these will have a long life, not all ML is about frontier LLMs.
I wonder if we will see an adoption of alternative floating point formats. IEEE floats are notoriously terrible at lower widths (<= 16 bits). Floating point formats such as posits do much better at 16 or 8 bits. If you could train at 16 bits per value instead of 32, and suffer a much smaller inaccuracy penalty than you would from IEEE32 to IEEE16...
For some reason I still haven't heard any predictions on when new fabs will come online to meet the current demand. This shouldn't be too hard to find out, since the building time of fabs is very predictable process.
The difficult question is more whether foreseeable memory demand will remain at the current level, grow even further, or shrink again.
Supply will not meet demand. What incentive do the handful of dram manufacturers have to end the party? This is what happens when legal monopolies finally win control. Dont't worry. The patents will expire in a few decades. Our grandkids will see DDR5 get cheap again. The system functions as intended.
Patents is not the issue here. Not even close.
The up-front investment of a memory fab is measured in billions, and takes years to construct and get running. The margin on the chips themselves is terrible, so without scale its not worth even trying. DDR5 is a industry standard that takes some effort to conform to, but the licence fees is a drop in the bucket to the cost of creating a fab.
The fabricators were cautious about increasing production, and slow to start planning. It takes further time to build up capacity, and if the demand drops down, they may end up producing dram at a loss when the market flips over to oversupply. The demand whiplash could kill any company that dared betting on increasing production. See the "bullwhip effect" https://en.wikipedia.org/wiki/Bullwhip_effect which has killed semiconductor fabricators before.
There is a discussion to be had about how to maintain national semiconductor production in Europe and US as a strategic industry, but historic attempts have all failed.
Billions is nothing in this market - if the market is supply constrained in the medium term then the hyperscalers will purchase their own route to manufacture (e.g. through coinvestment).
Also that's not what the bullwhip effect is - although I know what you are saying. The bullwhip scenario is about the effect of communication and batching through various layers in the supply chain, this is more similar to the cobweb effect/theory.
I have fairly simplistic view of the economics involved here. Could you explain why the ability to sell more chips wouldn't be sufficient enough incentive to increase supply?
Not the person you’re replying to, but RAM has historically been a boom-or-bust business, and companies that invest to meet demand during a boom cycle usually have that new capacity come online just in time for the bust.
If it was just variable costs and new capacity was available today they’d do it. But there are substantial fixed costs and delays to increasing capacity, and that uncertainty makes it risky.
That's such a nonsensical argument, it holds for every other business too and in this case it's just a lame excuse for monopolization. If you are that chicken and can't stomach competition you should not be in business anyway.
The current RAM manufactures were convicted of conspiracy to manipulate prices back in the 2000s or thereabout, doing so is their modus operandi, but this time the government is participating in the racket.
Chip manufacturing has unusually long spin-up times, high capital costs and relatively thin margins for anything but the latest and greatest processes, compared to most industries.
Bringing on new fabs takes many years and billions of dollars. You're exposing yourself to a lot of risk if you build now and find that the gold rush is over by the time your new capacity is online.
Let's imagine you're drilling oil instead. You have to spend billions of dollars over years finding and developing a new oilfield to make any profit back. And once you have it, you have to continuously spend enormous amounts of money to keep producing it, which means your effective price floor is higher than the current stable price.
Now it's 2021 and someone gets a tanker stuck in the Suez, sending the price of oil sky-high. How long does the ship have to be stuck before you spend those billions of dollars on a bet that it'll recoup before someone gets the ship out?
Although on the flipside, let's pretend it's 2017's and you are Nvidia selling GPU's for Bitcoin - maybe demand will dry up at some point? Do you stop scaling production as this might be the max of the market, or do you follow the market and increase production?
It's always easier to see the right move in hindsight!
Nvidia doesn't own fabs though, TSMC does. By 2017, ASICs for Bitcoin were well underway. Ethereum hadn't switched to PoS, and wouldn't until 2022. For that specific question, the answer is yes, because the GTX 1080 Ti is/was a monster card, and the crypto miners have a somewhat predictable demand for them, so there's some modeling you can do based on demand for the 2016 generation of cards. The question is ofc, if you're Nvidia, what are you optimizing for? Let's say, without foresight that Ethereum would move to PoS in 2022 and that AI would replace that demand, how many 1090 Ti cards do you make, how many 1070s, how many mobile 1080s, how many Titans? In order to answer that, someone at Nvidia would have to have, for better or worse, really had to have gotten into cryptocurrency in order to understand that market. Because you, as Nvidia, know how much better the 1080 will be for mining Ethereum, certain predictions can be made on demand.
Question is, without hindsight, 2022 rolls around, Ethereum moves to PoS, do you sell NVDA?
Its a lot easier to commit to spending billions of dollars in a hypothetical then reality.
> a path to a ~3x hardware cost reduction
Really?
How long do we have to wait until that ... cost reduction hits us?
For supply to meet demand. Depends very much on how aggressively producers scale and on how demand grows or shrinks.
Safe to say at least a year or two. It'd be shocking if it took a decade.
2-3x is completely dwarfed by the remaining improvements in training which is still in its infancy relatively
Probably, but at some point we're very likely to run out of significant training improvements and it's not clear that we'll see that point coming from a long way out.
Likewise it's probably dwarfed by improvements in how we make dram - continuing the roughly exponential (maybe a bit less recently) scaling of chips - but not necessarily.
The 2x from returning to previous costs is interesting because it's practically guaranteed, and it's on top of everything else. We're just currently "overpaying" (relative to the stable market price) for the manufacture of dram because of a sudden increase in demand.
my reply from the other thread fits here too:
> this is just not true at all, there are massive leaps from algorithms, data, etc. every year. scale is one axis of many and you need to get them all correct.
Unless there's a new paradigm, scaling up is all they can do to improve performance. They've shrunk down all the way to 1-bit models and all the low-hanging fruit is gone. There's no way for them to get much smaller, so they have to get bigger and faster to meet expectations.
this is just not true at all, there are massive leaps from algorithms, data, etc. every year. scale is one axis of many and you need to get them all correct.