https://annas-archive.li/robots.txt

https://annas-archive.li/llms.txt

robots.txt is a machine-parsed standard with defined syntax. llms.txt is a proposal for a more nebulous set of text instructions, in Markdown.

https://llmstxt.org/