I have recently formed an untestable hypothesis, which is that my similar (or stronger) resistance to this comes from having grown up in direct contact with mentally ill family.

In some ways, my theory of mind includes a lot more second guessing as a defense mechanism. At a foundational level, I know there can be hallucination and delusion that leaks out, even when the other party is in peak form and doing their best to mask it and pass as functional.