I don't see the issue. Attacking a physical lock as you suggest is analogous here to breaking the cryptography in some way. So the NSA busts out its sooper sekrit quantum computer ...

The inability to betray is a sharp observation but I think the analogy still holds flawlessly. It's a physical lock that you haven't handed out the key for so you're the only one with access to it. However the public counterpart still defies easy explanation.

Why did you put "symmetric keys" in scare quotes? Is that not the common term in your neck of the woods when speaking about symmetric crypto?