I ran into similar issues. Sometimes though there are good language specific solutions. For Rust based worktrees you should definitely check out https://mr-boxington.jdx.dev/ .

Regarding swapping, are you sure it's not just from ram backed tmpfs partition? For me that was it and most of the swap was from /tmp. So I created an agressive job that runs frequently to delete all tmp files not used in the last ~2 hours without any command holding handles to it.

The other thing I found was that there were a bunch of leftover processes from prior coding agents forking commands and just leaving them. So I had to build a system into my agent spawner to automatically launch each agent session in a cgroup whose processes all get killed on task completion.