In the few games I tried, it always ended up that way. You start with a few prime numbers (if you're lucky). Each time you click on one, it gets replaced with another number that usually isn't prime. After a short while, there are no primes left and then there's nothing to do but sit and wait for the game to end.

Perhaps all the numbers should have a finite lifetime, after which they evaporate and a random new one replaces them.