I agree but in these cases the genetic variance that is being accumulated, is only limited to a few individuals who more or less don't interact with the world, except through their workers. But the workers themselves have no way of accumulating or passing on the genetic variation.

In some sense the genetic feedback loop for ant population is designed in such a way that, it makes sense when looking at each ant colony as a singular organism

Unlike the giraffe, or elephants, who are individually capable of accumulating genetic variations.