It's even worse when you end up installing PyTorch as a separate package in some other layer. It's not shared between layers at all with regular Docker.