> It isn't trivial to tell if the weights have poisoned:
True, but it is easier if you have the weights than if you don't. I guess this was their point?
> It isn't trivial to tell if the weights have poisoned:
True, but it is easier if you have the weights than if you don't. I guess this was their point?
Yes that would be my view. Open access doesn't make it trivial, but remains the most expedient way to remediate these exploits.
You should assume the weights have been poisoned, the model has been prompt injected, etc and design software around it accordingly. You can’t really trust the model and it makes sense to always have controls around it and not depend on it behaving a certain way.
I’m aware this doesn’t happen, just saying. There are more examples of a model randomly hallucinating and deleting something than of a deliberate compromise.