I completely agree. You are infinitely more likely to get implicated in some widespread attack due to bugs in GitHub Actions or your automated release scripts than have your local machine's local build and signing infrastructure attacked.

I have yet to see any evidence that fancy CI/CD systems are better than good old fashioned tarballs and detached signatures. Bonus points for distribution packaging systems where they add an additional layer of review and separate validation of releases. People seem to gloss over that fact the "stodgy old-fashioned" rigamarole of Debian is part of the reason why the entire internet didn't pwned by the xz attack.

At the very least you should require a human to sign the blobs before the release is actually published. (This isn't always enough if the attacker can add themselves to the maintainers list and sign with their own key, which is why the distribution packaging systems where they maintain their own trusted copy of upstream keyrings is far more preferable.)

If your threat model boils down to "if my GitHub account gets attacked or even a single API key is leaked all of my users are fucked" then you really need to take a long look at a mirror and ask yourself if that is reasonable.