So far all the Kimi models have open sourced their code, their weights, and published technical reports explaining their training methodology. I expect the Kimi K3 Technical Report will come out July 27 and they usually publish the code and the weights alongside that.
clearly not true. the weights are the preferred form for making modifications. Do you really think people should be downloading hundreds of TB of training data and running make to build the model on their own cluster of GPUs?
Random people? No. Governments and big corporations? Yes. It removes any concern of "backdoors", and is currently the best starting point for your own model which will be as capable as k3.
I want to caution against this line of thinking. NSA's fast16 program silently altered data during nuclear simulations, and that is also possible within open weight models. There is no reason to think there is anything like that currently, it also cannot be dismissed. Open source would include the training data so you could create a similarly capable model.
Training runs are not deterministic. Not just in the case of floating point not being associative, but nodes have hardware issues and go down and back up at various points through training. Problems get encountered and then various parameters get changed during the middle of a training run.
Making the process of creating the source code repeatable goes beyond the idea of open source. Open source is about being able to work with the code, not recreate it. It doesn't require domain experts to document every single thing they know. For example look at some of the GPU drivers in the Linux kernel. The GPU is not properly documented and there is trust that vendors are implementing things correctly.
>You can also use a hex editor to modify compiled binaries
It would be like if someone gave you a prompt you could pass into ChatGPT to produce the entire Linux kernel. While yes you could modify that prompt and then spend a ton of money on inference to generate millions of lines of code which hopefully are equivalent to the original Linux kernel it would be easier if you just had the source code to work with where you can make a simple extension. If you want to end a model you want the weights, and not the entire setup to generate it. The weights are the starting point that you can work off for training a new model similar to how the source code is the starting point people want to work off of.
Kimi publishes their code and a technical report on their methodology. But I think the weights are still important. It means anyone with the resources could run the same model on their own hardware.
The training corpus + training code is just an automated editor for the weights. It would be like requiring the source code for Visual Studio for software made within it to be open source.
Weights are not an output artifact no more than source code is. There is never a moment where you can claim that it's done. As requirements change new ways to change the weights / code come up. With different projects you might want to import the weights / code into a bigger model / codebase.
So far all the Kimi models have open sourced their code, their weights, and published technical reports explaining their training methodology. I expect the Kimi K3 Technical Report will come out July 27 and they usually publish the code and the weights alongside that.
The weights are the "source" of a model.
If weights are the source for models then ELF binaries are the source for software.
Everything is open source if you know how to reverse engineer ;)
Then everyone is open source if you have $20 to spend on tokens, I guess.
clearly not true. the weights are the preferred form for making modifications. Do you really think people should be downloading hundreds of TB of training data and running make to build the model on their own cluster of GPUs?
Random people? No. Governments and big corporations? Yes. It removes any concern of "backdoors", and is currently the best starting point for your own model which will be as capable as k3.
I want to caution against this line of thinking. NSA's fast16 program silently altered data during nuclear simulations, and that is also possible within open weight models. There is no reason to think there is anything like that currently, it also cannot be dismissed. Open source would include the training data so you could create a similarly capable model.
Training runs are not deterministic. Not just in the case of floating point not being associative, but nodes have hardware issues and go down and back up at various points through training. Problems get encountered and then various parameters get changed during the middle of a training run.
Making the process of creating the source code repeatable goes beyond the idea of open source. Open source is about being able to work with the code, not recreate it. It doesn't require domain experts to document every single thing they know. For example look at some of the GPU drivers in the Linux kernel. The GPU is not properly documented and there is trust that vendors are implementing things correctly.
> Training runs are not deterministic.
I agree which is why I said you could make a similarly capable model.
> Open source is about being able to work with the code, not recreate it.
You can also use a hex editor to modify compiled binaries, but no one would consider that "open source".
>You can also use a hex editor to modify compiled binaries
It would be like if someone gave you a prompt you could pass into ChatGPT to produce the entire Linux kernel. While yes you could modify that prompt and then spend a ton of money on inference to generate millions of lines of code which hopefully are equivalent to the original Linux kernel it would be easier if you just had the source code to work with where you can make a simple extension. If you want to end a model you want the weights, and not the entire setup to generate it. The weights are the starting point that you can work off for training a new model similar to how the source code is the starting point people want to work off of.
The weights are the output artifact, the training corpus and system are the source.
Kimi publishes their code and a technical report on their methodology. But I think the weights are still important. It means anyone with the resources could run the same model on their own hardware.
The training corpus + training code is just an automated editor for the weights. It would be like requiring the source code for Visual Studio for software made within it to be open source.
Weights are not an output artifact no more than source code is. There is never a moment where you can claim that it's done. As requirements change new ways to change the weights / code come up. With different projects you might want to import the weights / code into a bigger model / codebase.