Even more than that, a lot of vulnerabilities require untrusted input to exploit, but the library or executable is usually only used with trusted input. For example, suppose the chacha20 function had a vulnerability that was exploitable with a malicious secret key, but you only call it with your own, definitely not malicious, key. The vulnerability doesn't impact you, but your security policy might still require you to update. And no amount of splitting the package will help with that.
I have concluded that even though I won't use that key, I still want the issue fixed. You never know if the key I randomly choose next year is the vulnerable one. Worse, you never know if someone else won't make a bug that allows outsiders to select a key...
But it doesn't warrant an emergency automatic update.