Absolutely. Look at the Portuguese Empire: even at the end most Portuguese were poor and many had to emigrate.

Most British had a low standard of living going into the XX century. Engels' report to Parliament painted a bleak picture of the standard of living in Britain at the peak of the British Empire.