Think of the models making progress on CIFAR-10, ImageNet, CelebA, etc. 15 years ago. They had issues too and weren't just scaled-up as is to the architectures we have today.