That doesn't help if my machine (with only a few USB ports) gets stolen/lost with the token in it. It doesn't help if some of my devices only have USB-C and some only have USB-A. It's absolutely more annoying than letting my password manager fill things in or typing in a 6 digit code from my authenticator app.
Get a better password manager? Most store passkeys.
If the passkey can be stored in the password manager, then there's no second factor and what's the point?