That's why I build project specific images in CI to be used in CI. Running apt-get every single time takes too damn long.

Alternatively you can use a local cache like AptCacherNg