I have been writing bots and scrapers for almost 20 years.
Not once have I had someone I work for in this capacity even ask about robots.txt
I have been writing bots and scrapers for almost 20 years.
Not once have I had someone I work for in this capacity even ask about robots.txt
How much would I have to pay you for you to change careers permanently?
It wouldn't make much difference. Me, I just do it for fun.
robots.txt can used as a reward mechanism. If an automated visitor that we would normally block begins by accessing robots.txt that's a good sign and we give that a plus score or whatever. If subsequent accesses from that visitor comply with robots.txt, we let them continue (perhaps with some rate checks).
Conversely, if an obviously automated visitor does not fetch robots.txt, that can be an instant ban, kind of like a reverse honey pot, if you will.
So it can have some nonzero value in the overall context; it's just silly to think of it as a blocking mechanism.
In short, robots.txt documents what access is allowed; it states the policy. Enforcing it the job of something else.
Someone you work for in this capacity shouldn't ask about robots.txt because that's the sort of detail for which they have you to deal with.
You are part of the problem, sounds like.
> You are part of the problem, sounds like.
And may be or could be part of the solution by them taking upon themself to implement it in an ethical manner by having the bots and scrapers respect robots.txt; unless the spec specifically says not to in that case I agree with finding new clients or another job
This implies the solution is every actor in the world willfully complying which doesn’t sound very realistic if you ask me.
Similar to saying world hunger would be solved in everyone who could donated $100 to those in need. Sure it may be true but foot stomping about why isn’t it organically happening is missing the point.
There is a clear difference between the global issue of famine and actively choosing to work for an arsehole corp.
The problem itself does not matter. If the nature of any solution relies on willful global compliance, it’s not going to be very effective at whatever it’s trying to solve.
Every society fundamentally relies on willful compliance. Sure, you can force a few outsiders, but once a sizeable population stops being compliant, it just breaks down.
Hence why the internet doesn't work well.
The internet is not a society. The internet is all societies. Hell, more and more it's not even human societies.
Denied. One can choose not to be an active element in that problem.
Yay, you solved murder, give yourself a good pat on the back.
You're saying that murderers shouldn't be blamed for murdering?
I'm saying telling people they shouldn't do something is about as useless as farting in the wind. Blaming someone does nothing, it's the force of action behind the words that mean something.
But, because this person could be anyone, anywhere your words are hollow and empty. You are impotent to stop their future actions.
So, what’s your proposed solution?