In the case of kernel.org, why not make the unauthenticated version return only the latest kernel repo with no history (tiny number of URLs relatively speaking). If you want full kernel.org features, login.

because they do not want to be twitter, reddit, facebook, ...

I'm not suggesting 'login' because facebook/twitter/etc, just as a mechanism to make the bot problem go away. They clearly want it to stop, they tried obvious methods but the AI platforms are circumventing it (deliberately) which is poor form.

login or you will not get the full service is exactly what facebook and co are doing and what your suggested would amount to. the kernel.org stated quite clearly they want to remain a public service. which i applaud.