You should assume the weights have been poisoned, the model has been prompt injected, etc and design software around it accordingly. You can’t really trust the model and it makes sense to always have controls around it and not depend on it behaving a certain way.

I’m aware this doesn’t happen, just saying. There are more examples of a model randomly hallucinating and deleting something than of a deliberate compromise.