This would be interesting on a Spark or two. I don't know how seamless the GPU and networking cluster setup would be when resource sharing between agent and post-training VMs.

Unfortunately as of today DGX Spark doesn't support GPU PCI passrhru for KVM

A good alternative is incus and running LXC and or OCI containers, GPU works in those and can be shared with multiple instances