All the gold? Certainly not. That's just a meme at this point.

For example, Brazilians love to say that the Portuguese stole their gold. Even the highest estimates I could find amount to like 1000 tons of gold which works out to around 135 billion dollars today. Meanwhile, the government robs them of several times that amount every single year: tax revenue is currently around 815 billion dollars.