Same here. My Yubikey-equiped M1 MBP also refuses to sleep, so I end up shutting it down every day. I suspect it appears as keyboard input periodically waking the machine.