For capabilities, a special instruction only makes sense in the context of CPU memory access and this is all about out of bounds C bugs. OS capabilities do not need new instructions.
And then, out of bound memory access may be better solved with better programming languages, thought of course we need to live with the legacy code.
I think what the parent means is we should be able to create syscall sandboxes within the same process (like a library not being able to do IO). Maybe I'm wrong but I think this could sort of be implemented with CHERI, by restricting syscalls to the official libc entry points (like OpenBSD) and requiring a capability pointer to access the functions.
.NET Framework tried that with their whole "security" system, but it was a massive failure.
The only fool-proof solution is separate address spaces and OS cooperation.
In regards to bounds checking, that has been solved in Solaris since ADI was introduced in 2015.
https://docs.oracle.com/en/operating-systems/solaris/oracle-...
Language safety helps get you 80% of the way there, but you are still working from software on top of fundametally unsafe hardware. Companies and agencies are and, increasingly, will pour money into hardware that give certain safety and security guarantees.