I don't agree, but I think your perspective is valid so I upvoted you.
I've worked for companies that did this, in fact I'd say this is commonplace at startups and small companies, at least till around 500 employees or so. If I ever start another company, I'd operate the same way as a founder/ceo. While Dunbar's Number probably puts an upper limit on the number of people I can truly know in my organization, I think it's incredibly important as a leader to have the possibility of putting a face to a name inside the one endeavor (company) you are wholly responsible for. That doesn't say anything about my trust or lack there-of in the people below me on the org chart.
I almost feel the opposite. I've turned down jobs because I want to understand the strategic direction and the folks I was interviewing with couldn't adequately explain it and they were unwilling to let me meet the folks high enough up the chain that they could as part of my interview process. If I can't get the time of day out of the founder during one of the most important decisions (hiring) that their company makes, they're certainly not going to give me the time of day once I'm in the door and that means I have effectively no feedback mechanism. Why would I want to work there?