The issue is there's far more pages than could reasonably be stored: cgit is an old-school server-side renderer and there's a combinatorial explosion of possible pages it could render. The article calculates there's about a quadrillion different pages that could be 'crawled'
Most of those pages are going to be identical if not for some very minor differences. You can make it much much cheaper to put together these pages as opposed to the expensive operations it has to do right now.