Interesting paper, and something I have been thinking about. I am retired so I am a lighter user of AI than most people here but I still have Gemini and ChatGPT to a half dozen deep research studies for me a week. It is sobering to see how many web sites are speculatively searched. I mostly find the results useful and I prefer this new process to manual web search. After deep research asking for 'the best' reference link usually produces something else worth reading in addition to the research report.

Someone else here recommended sites maintaining their own CLAUDE.md file, good idea but too vendor specific. Ten months ago someone online was recommending the name llms.txt as a generic markdown file for agent use and I added one https://markwatson.com/llms.txt I stopped collecting web page visit statistics so I have no idea how often that file is discovered however.