The resason is that only Google bothers to put enough hardware security features to build software on top that allows to make a really secure device that blocks tampering.

That's not a reason. When the hardware doesn't have those "security features", then don't "really secure", just run without being "really secure".

I never treat my (Android) phone as secure anyway.

Security is GrapheneOS's raison d'être. If you don't want security, you can run another Android build that does run on the hardware you have.

Well, they're not improving people's security overall by limiting themselves to Google's hardware.

Thats are improving peoples security overall by only using pixels at this time. Nothing else currently meets the hardware security requirements except pixel devices. To use subpar hardware would set the security back substantially. They are not going to compromise on security for the sake of broad device support.

Also note that Motorola Mobility has stepped up to provide the needed hardware security features. So it will expand to a subset of Motorola devices next year.

Their goal is to hold a standard of security and privacy that can be trusted. They want the brand to be associated with that level of security and privacy. Sticking to Pixel allows them to leave their brand undiluted in this respect.