Don't ion drives consume comparitively a lot of xenon to work?

Yes, they have substantially lower specific impulse than fission fragment.