Took me some thinking to figure out where this design is not optimal.
The first step is to separate the problems. The content addressable object storage and storing the refs. libgit2 has two backends for those already.
Then use appropriate distributed dbs for the two backends. With large scale something like Cassandra for the objects (AP) and FoundationDB (CP) for the refs (or the equivalent Amazon services).
For performance the objects need to be cached locally and requests routed to nodes where the objects are cached locally.
Part of the problem is that the API you're supposed to provide externally depends on git packs which you might have to reconstruct on the fly from such a system.
Exactly. I would love to see that design work, but it doesn't for any realistic workload.
Then the local cache system needs to provide or cache those git packs