What you are describing is basically remote buildkitd. That allows all of your docker builds to share a big cache. The cache-to/cache-from approach is of limited usefulness.