Yeah, I know what you mean. I do my best to check myself and also run things by people I trust, but there's an ever-present risk I'm going insane.
As a test, I've been attempting an incredibly complex project that goes far beyond my abilities as a kind of deliberate worst-case-scenario. It's more or less a programming language for a very specific purpose that compiles to a custom bytecode and runs on a custom runtime with specific performance guarantees.
I've spent part of the last month iterating on a formal model of the system and various specifications. Along the way, I teach myself how to understand and critique the part of the system I'm working on, however I also deliberately keep things just beyond my understanding by opportunistically pulling in concepts from various sources ... algebraic topology, obscure corners of PL, concepts plucked from similar systems. It's a complete monstrosity with, now, hundreds of supporting documents, research spikes, processed references, critique passes, etc.
If I'm able to complete this project and have it work as expected, I think I'll have learned a lot about what is or isn't possible. If the current design does in fact work, I'm fairly confident I'll have advanced the state of the art in the niche field I'm working in.