> the notoriously insecure Intel Secure Enclave being the only protection

While I share your concerns about Intel SGX, your statement is not exactly true: SGX is only meant as an additional measure to secure insecure PINs.