I'm in favor of open source AI... however:
> It's theoretically possible for a bad actor to embed hidden adversarial behavior in a model. But if this happens, it serves the interests of responsible actors to find these exploits as soon as possible, and the best way to do this is to let anyone who wants to inspect them.
This is a bad argument. It isn't trivial to tell if the weights have poisoned:
``` https://www.thedeepview.com/articles/microsoft-how-to-spot-a... https://futurism.com/future-society/easy-poison-open-weight-... https://semgrep.dev/blog/2026/ai-supply-chain-problem/ ```
I'm not arguing in favor of closed-AI; I'm simply saying poisoning may be subtle.
I was gonna say that at least the frontier labs may be motivated to not-poison their own models, but then Anthropic just attempted to poison Fable's LLM training capability so... sigh. I'll try not to derail.
It's also not trivial to tell if an open source project has had malicious code added. It might be a bit easier than to tell if there's malicious code than if the weights in a model were poisoned, but for 99% of users, you're going on pure trust in both cases.
And in both cases, even this seems better than relying on a closed source, service only solution where the same issues could be completely undetectable (at least without way more analysis)
> going on pure trust
The trust in open source software is founded on the process which involves people being able to understand the code given they have the right expertise. I don't see how this process works with open weights.
> It isn't trivial to tell if the weights have poisoned:
True, but it is easier if you have the weights than if you don't. I guess this was their point?
Yes that would be my view. Open access doesn't make it trivial, but remains the most expedient way to remediate these exploits.
You should assume the weights have been poisoned, the model has been prompt injected, etc and design software around it accordingly. You can’t really trust the model and it makes sense to always have controls around it and not depend on it behaving a certain way.
I’m aware this doesn’t happen, just saying. There are more examples of a model randomly hallucinating and deleting something than of a deliberate compromise.
Exactly. Faced with a choice between open weights that might be poisoned and walled-off weights that I know are poisoned... well, it's an easy choice.