I think what's actually needed is two things: an EU training infrastructure that allows training of 10T+ models, and an EU inference infrastructure that is sufficient that it's possible to do RL on them.
This effectively reduces the problem to a specialized supercomputing infrastructure problem which I think is relatively easy to solve. I think the chips are coming. I think Euclyd will be able to do the inference chip and I think the training chip won't be harder. It's just a matter of accepting the need to order a huge number of them, being willing to think a little bit like the kind of people who operate corners. So we can be there next year, I think. What we then lack is a training chip-- maybe OpenChip can do it, maybe they can't, but there are reasonable but still unfinished projects. Maybe if Euclyd finishes an inference chip in 2027 we can have the state pay them to make a training version, put in fp32, put in communication tiles. If their design is real and works (which it should, since it's basically a fancier version of Groq, as it's described, and since even Groq works) I think the advantage these chips is likely to have would be enough that a training version would be NVIDIA-beating.
We probably need some solution for the data-- i.e. to allow people to do things that are against copyright law in a limited way, but I think it's a better idea to start EU firms than to try to attract Anthropic.
Because of the need for capital the hardware-software carousel is necessary. We can't pay for NVIDIA chips and then have NVIDIA feed that money into US firms. We have to feed money into EU chips that either carousel the money into EU AI firms or who just offer cheap chips.
The big question is who is going to fund it? Is it tax payers money? If so how can they guarantee it’s not going to be another waste and corrupted disaster. Also EU is already late to the AI race and by the time the lawmakers starts to think about this, it would be game over
> The big question is who is going to fund it?
this feels like a small question, the money needed for this isn't that much (probably ~20b) and every country needs cyberwarfare abilities, especially automatable ones, so the military can pay for it. And its not like 10T models cant be used for anything else, so people who want ROI in health, chip development, all sorts of stuff
That’s assuming everything would be in time and no budget overflows which is never the case in public funded projects.
I can give you one concrete example. Google for Stuttgart 21. It’s a project to build a main train station in Stuttgart, Germany. The project has been delaying for years. By the time it’s finished it would be 12.5 years overdue its original deadline and so far it has costed 14.5 Billion euros.
One could assume that the biggest economy in Europe can build a train station in no time because how hard could it be? Given the fact that they can’t even do that, I can imagine how it would go with publicly funded cutting edge technological AI endeavour. Even if EU has billions of euros to burn, what they don’t have is time.
train stations are boring and no military spends time thinking about train stations. ASI hacker nonsense, that gets their attn.
I think the chips alone are 10B minimum. It'd be way bigger than CERN.
Provided that the systems work, they can at least be repurposed to other things. If the organizations that are to train public LLMs can't do it, we can rent the system out to Mistral or something.
So I think something like 5B, starting with 10B to get started, in public money per year, the chip firms are private, some of the LLM firms will be private, but the system is available to train European LLMs-- that's I think a realistic approach.
The biggest problem with public money funded endeavours is corruption. At the end of the day most of the people would be there for the pay check and even if nothing is delivered after burning billions of euros, nobody would be held accountable.
That’s why I think it’s not going to be successful with public funding. What EU needs is structural unpopular reforms. Reduce taxes so companies and top talent would have an incentive to choose EU over US. Reform employment laws so people can be fired for being pencil pushers and lacking off. Relax privacy and copyright laws so the data can be used to train the model. Completely repel all laws that create a bureaucratic nightmare for startups. Today all these suggestions would be a complete no go in any European country, but that’s the hard reform EU needs like yesterday even to have a chance to compete against US and China.
Yes, of course people would be working there for the paycheck.
I think there's nothing special about public funding though. The field is so competitive that people will be mad if a competitive model is not achieved, making corruption more damaging to the organizations. There would also be some internal competition. There are after all several EU LLM/AI/etc. firms that would probably try to use this infrastructure.
By paycheck I meant that they get the money through corruption. Like fake bills, bloated estimates and huge commissions, but in the surface it all looks legal and they get away stealing millions of euros of public money of hardworking tax payers.
I guarantee you that maybe people would get mad that no competitive model is achieved because the politicians burned the tax payers money. But there would be no repercussions, nobody would be held accountable and would just be marked as a failed project.
I think the bigger issue of public funding and corruption, is that usually these contracts are awarded to "friends" with the purpose of 100% mismanaging and stealing as much of the funds.
Well, some risk must be taken, even the risk of such things.
Some sort of alternative must be created, after all, since models are being restricted to the US only.
Yes I agree, I'd only wish there's some way for do this by the book, like its not so hard, Brasil government has produced almost SOTA models, why cant the UE figure shit out? They could even rent hw like everyone else to do the first trainings...
But no, lets make this a 10B deal who someone will suck out 1B out of it even before its shipped
I think the way to do it is this: let EU chip design firms bid. They say what their systems can do, they give their prices, and then we choose the one that can achieve the requirements (pretrain multiple 10T+ models in a reasonable amount of time and then do RL on them) at the lowest cost.
There has to be a deadline and also a penalty if they don’t deliver. That way the company cannot just take the money and pocket it and just say oops, we failed
GLM 5.2 is ~40B active parameters, which is what matters most for training cost.
Yeah, but if its final performance comes from being trained with data from a bigger model one can question whether it's a way to build genuinely new 40B models.
for RL cost*
pretraining becomes more expensive actually as you make MoE models sparser (you need more tokens in the pretrain, and if you don't have that then you need to train for longer)
Europe has asml
Yes, but what does that affect?
If we end up with a world where only US firms can use the latest LLMs and the latest LLMs are needed to keep up in the software world, or in making prototypes, then that's a whole series of fields which are blocked from us.
So I think we need to make sure that we no only can, but build frontier LLMs from scratch-- not RLed on foreign data, but genuinely from scratch.
Even from an economic point of view, I don't think a continuous outflow of ~200 USD/month for every office job is sustainable, and that's what we'd get if the plausible scenario is borne out. An inter-EU cost of 200 USD/month for every office job though, that's survivable.
asml is part of the chip supply chain; amsl has chip making talent, and EU needs more better chips to make more better ai
also an em-dash and genuinely in the same sentence? cmon bro how much claude have you been using?
A lot actually. Obviously not to write these comments, but a whole lot of Claude.
With regard to the substance: sales of ASML machines are not currently connected to the EU getting chips, but to ASML getting paid for their work. For EU chips for training transformer models we'll need chip design firms, not chipmaking, and as I stated in my comment, there are some promising ones that will probably be able to design the chips we need if we order them.