luckily nobody made an actor library in the most pupular programming language that can bootstrap a (resident) remote process in one line of code. it would be a shame if someone did that and then also build a made tool calling process of the harness installable on everything with a stdio.
it's not like it's any worse than just giving the thing access to your ssh keys.
Agents and harnesses don't get access to "my" ssh keys, they get access to new ed25519 key pairs created for specific projects and access to discrete things. The blast radius is relatively well contained to specific VMs they are SSHing into for project specific purposes. I don't run a harness or agent directly on my personal workstation.