The stumbling block we have is spinning up separate environments for every agent so they have isolation for their branches. I think this is solveable, but we aren't trying to solve it ourselves. In practice it means we aren't doing a lot of agent supervision.
Git worktrees essentially solve this. It essentially copies your repo to a new folder
That sounds like an excellent match for containers.