I thought finetuning data can't contradict foundation models, and anything that are inconsistent with the standard LLM American-Chinese split personality would be rejected?
But you have to have more data than used for pretraining for the added knowledge to take precedent over pretraining, no? If that would be the case, you practically contradict the knowledge in the base model.
I mean ... LLMs are sort of an extreme and living proof of linguistic determinism. Their behaviors are dictated almost entirely by disorganized language data, primarily English and Chinese. So you can't just add a language as native primary language in a quick post training, I think. There's no way that it would work.
Funny, that's what I thought when PewDiePie set up his monster AI rig and what he called a 'council'. Quote:
"PewDiePie has built a custom web UI for self-hosting AI models called "ChatOS" that runs on his custom PC with 2x RTX 4000 Ada cards, along with 8x modded RTX 4090s with 48 GB of VRAM. Running open-source models from Baidu and OpenAI, PewDiePie made a "council" of bots that voted on the best responses, and then built "The Swarm" for data collection that will become the foundation of his own model coming next month."
Calling a bunch of LLMs a "council" is just rebranding well known ensemble methods with shrill marketing hype, nothing original or out of the ordinary. Mixture of experts and every other idea in his stack has a literature older than PewDiePie's career.
Yet another attention craving influencer who shilled crypto scams during the crypto bubble and is now marketing "AI councils" during the AI bubble.
He's not a serious or honest person, AI is just what he pivoted to after crypto. That's not innovation; it's attaching trendy branding to ideas that were already old when Marvin Minsky wrote The Society of Mind in 1986, three years before PewDiePie was zero years old in 1989.
The only thing PewDiePie's brought to the table is cleverly optimized YouTube thumbnails designed to attract clicks. The architecture is decades old; only his marketing and shilling is state of the art.
This gets better short-term results for a fraction of the cost, for sure, but what do you when China places an export control banning the release of open weight models? If you don't have your own talent, you're then relegated to using a base model from 2026 or whatever the cutoff date is, forever. That defeats the purpose of a 'sovereign' model made for and by your people.
I thought finetuning data can't contradict foundation models, and anything that are inconsistent with the standard LLM American-Chinese split personality would be rejected?
Fine tuning happens on top of pretraining, so of course it can "forget" pretrained defaults when warranted by the new data it's being fine tuned on.
But you have to have more data than used for pretraining for the added knowledge to take precedent over pretraining, no? If that would be the case, you practically contradict the knowledge in the base model.
I mean ... LLMs are sort of an extreme and living proof of linguistic determinism. Their behaviors are dictated almost entirely by disorganized language data, primarily English and Chinese. So you can't just add a language as native primary language in a quick post training, I think. There's no way that it would work.
They could apply the Polder Model of consensus decision making with a mixture of experts.
https://en.wikipedia.org/wiki/Polder_model
Funny, that's what I thought when PewDiePie set up his monster AI rig and what he called a 'council'. Quote:
"PewDiePie has built a custom web UI for self-hosting AI models called "ChatOS" that runs on his custom PC with 2x RTX 4000 Ada cards, along with 8x modded RTX 4090s with 48 GB of VRAM. Running open-source models from Baidu and OpenAI, PewDiePie made a "council" of bots that voted on the best responses, and then built "The Swarm" for data collection that will become the foundation of his own model coming next month."
https://www.tomshardware.com/tech-industry/artificial-intell...
Calling a bunch of LLMs a "council" is just rebranding well known ensemble methods with shrill marketing hype, nothing original or out of the ordinary. Mixture of experts and every other idea in his stack has a literature older than PewDiePie's career.
Yet another attention craving influencer who shilled crypto scams during the crypto bubble and is now marketing "AI councils" during the AI bubble.
He's not a serious or honest person, AI is just what he pivoted to after crypto. That's not innovation; it's attaching trendy branding to ideas that were already old when Marvin Minsky wrote The Society of Mind in 1986, three years before PewDiePie was zero years old in 1989.
The only thing PewDiePie's brought to the table is cleverly optimized YouTube thumbnails designed to attract clicks. The architecture is decades old; only his marketing and shilling is state of the art.
Lighten up dude.
This gets better short-term results for a fraction of the cost, for sure, but what do you when China places an export control banning the release of open weight models? If you don't have your own talent, you're then relegated to using a base model from 2026 or whatever the cutoff date is, forever. That defeats the purpose of a 'sovereign' model made for and by your people.