You keep building on the last available version? Fine tuning is a whole lot cheaper, easier and more useful than pretraining a model from scratch. It's a complete no brainer.
> But good to have researchers and professors "in-house".
I'm not in this field, but I think we already have them. Probably the main difference is that we have most or all of them in academia and next to none insode private companies. But we do have them, and they could start working for private companies if the market moves in that direction in the EU as well
Unfortunately in this game first principles requires massive resources, not "some". Building in-house on top of existing open weights is a good way to bootstrap this process, especially since there's nothing inherently magical or particularly expertise-heavy when it comes to weights themselves
You take current version and build on top of it. You have the weights.
You might not get some n+1 version at some point but the n version you will have will be still most likely much better than whatever you come up with burning good will money of people believing in „sovereignty”.
You are not getting ahead in this game by being „true to your local values” capital expenditure is insane in this game.
It seems like you don't understand. For fine tuning, it's cheaper to fine tune an existing model. For massive changes, it's better to retrain from scratch. Otherwise, model will UNLEARN a lot first, and then you will train about twice longer to the same result.
You keep building on the last available version? Fine tuning is a whole lot cheaper, easier and more useful than pretraining a model from scratch. It's a complete no brainer.
> You keep building on the last available version?
yes but a sovereign can allocate some resources and a few people to stay in the loop from a first principles level. No need to wait for a rug pull.
Of course, it can not compete with the frontier labs. But good to have researchers and professors "in-house". LLMs are here for the long-term.
> But good to have researchers and professors "in-house".
I'm not in this field, but I think we already have them. Probably the main difference is that we have most or all of them in academia and next to none insode private companies. But we do have them, and they could start working for private companies if the market moves in that direction in the EU as well
Unfortunately in this game first principles requires massive resources, not "some". Building in-house on top of existing open weights is a good way to bootstrap this process, especially since there's nothing inherently magical or particularly expertise-heavy when it comes to weights themselves
Seems like you don’t understand.
You take current version and build on top of it. You have the weights.
You might not get some n+1 version at some point but the n version you will have will be still most likely much better than whatever you come up with burning good will money of people believing in „sovereignty”.
You are not getting ahead in this game by being „true to your local values” capital expenditure is insane in this game.
It seems like you don't understand. For fine tuning, it's cheaper to fine tune an existing model. For massive changes, it's better to retrain from scratch. Otherwise, model will UNLEARN a lot first, and then you will train about twice longer to the same result.
https://en.wikipedia.org/wiki/Catastrophic_interference