Interesting that you would have such an uncharitable takeaway from this given you are a fan of his.
He uses it (the drug cocktail) as an example on understanding and its role on evaluating alignment.
He poses this question
>Before injecting this cocktail into your bloodstream, would you find it reassuring to know that there is at least one human cancer expert who understands – even partially – the mechanism behind this cure?
To which I would answer 'well yes of course'.
And I believe that you would too. Greater understanding is a good thing and worth striving for in a changing landscape.
That is what he is advocating for within mathematics. Whether or not we currently understand the mechanism of action of Pepto or Tylenol is not really relevant to the core message.
I'm not sure I'd find it that reassuring if "there is at least one" human cancer expert etc etc. That's part of the fallacy. One expert wouldn't give me too much additional confidence vs a super powerful medical AI. If we want to go the "reassurance by human" route, I think I'd appreciate if a significantly large group of experts understood it. But he didn't use this argument because we don't even have that in Math, where some proofs are validated by a tiny group of experts.
But if we really want reassurance, I'd want to see drug trial data reviewed by the machine and a group of clinical trial experts.