> robots.txt. This is not the law
In Germany, it is the law. § 44b UrhG says (translated):
(1) Text and data mining is the automated analysis of one or more digital or digitized works to obtain information, in particular about patterns, trends, and correlations.
(2) Reproductions of lawfully accessible works for text and data mining are permitted. These reproductions must be deleted when they are no longer needed for text and data mining.
(3) Uses pursuant to paragraph 2, sentence 1, are only permitted if the rights holder has not reserved these rights. A reservation of rights for works accessible online is only effective if it is in machine-readable form.
> A reservation of rights for works accessible online is only effective if it is in machine-readable form.
What if MY machine can't read it though?
That’s your problem.
A solution has been offered and you can adhere to it, or stop doing that thing which causes problems for many of us.
I doubt robots.txt would fit. robots.txt allows or disallows access, but it does not state any claim. You can license content you don't own, put it on your website, and then exclude it in robots.txt without that implying any claims of rights to that content.