The throughput quoted here is for a single core, and it is not reported any attempt to measure how well this scales to all cores.

A modern Zen 5 should reach a throughput more than twice this value per core, with a total over 3 TFLOP/s for the complete CPU.

That 100 TFLOP/s for FP32 is for a GPU that might cost from 30 to 100 times more than a desktop CPU, so it is not certain that its performance per dollar is any better than for the desktop CPU.

This is very different from 7 to 10 years ago, when GPUs had a far higher performance per dollar than any CPUs. Since then, the performance per dollar of the desktop CPUs has increased, mainly because their prices have not increased much, while the performance per dollar of the GPUs has decreased, mainly because of a great increase in their prices, especially for the "datacenter" GPUs, which now may be more than 10 times more expensive than they were 7 years ago.