I’ve spent most of my career in Job Tech and HR Tech so I can give you some insight.
LinkedIn lost a lawsuit about specifically trying to prevent scraping of their job content. This was a big deal at the time as all the major job sites were scraping jobs either because a customer lacked technical capabilities or to add jobs that were not part of the corpus. It’s a routine part of how the industry works in the same way that not many complain about Google scraping the Internet.
Secondly, and actually more importantly, a lot of the industry exchanges jobs using feeds. This is where CPC job distribution generally exists. These can contain “free” jobs which do not get any CPC credit but would let you build something like this site without any scraping infrastructure. You can ask some major job boards for just the free jobs if you wanted that for some reason. Most job specific scraping services like Aspen and Feedonomics will deliver you scraped jobs in the your preferred feed format.
In practice where I’ve worked we just blocked scraping sites when we get a complaint and respected robots.txt. It was rare for someone to complain since we were good sources of traffic wherever I worked. I am not a lawyer but my understanding is that as long as you’re not otherwise breaking the law by respecting legitimate takedown notices then scraping is fair use.
Thanks for sharing the industry insights! Very cool to learn about CPC job distribution, etc.
I have also read the LinkedIn vs. hiQ Labs lawsuit. The ruling is significant because the court finds that scraping public data did not violate the CFAA, though it violated LinkedIn's tos. LinkedIn ultimately wins at the end because hiQ was bankrupted. One of my take away is to scrape responsibly by rate limiting the requests and not overloading the server, etc
The goal of the companies and recruiters is to fill open positions with qualified candidate asap, hence why some would even pay job boards to promote their jobs or offer referral bonus. So my take is that companies would love job boards like this to freely advertise their positions to folks who might not otherwise come across these positions and increase their top of funnels. So in this case, the incentive of aggregators and companies are aligned and it is a win-win situation for both.
I don't know why a company might not want to partake in this, but would be happy to take it down upon their request if they like to make it more difficult for candidates to stump upon their openings.
I’ve spent most of my career in Job Tech and HR Tech so I can give you some insight.
LinkedIn lost a lawsuit about specifically trying to prevent scraping of their job content. This was a big deal at the time as all the major job sites were scraping jobs either because a customer lacked technical capabilities or to add jobs that were not part of the corpus. It’s a routine part of how the industry works in the same way that not many complain about Google scraping the Internet.
Secondly, and actually more importantly, a lot of the industry exchanges jobs using feeds. This is where CPC job distribution generally exists. These can contain “free” jobs which do not get any CPC credit but would let you build something like this site without any scraping infrastructure. You can ask some major job boards for just the free jobs if you wanted that for some reason. Most job specific scraping services like Aspen and Feedonomics will deliver you scraped jobs in the your preferred feed format.
In practice where I’ve worked we just blocked scraping sites when we get a complaint and respected robots.txt. It was rare for someone to complain since we were good sources of traffic wherever I worked. I am not a lawyer but my understanding is that as long as you’re not otherwise breaking the law by respecting legitimate takedown notices then scraping is fair use.
Thanks for sharing the industry insights! Very cool to learn about CPC job distribution, etc.
I have also read the LinkedIn vs. hiQ Labs lawsuit. The ruling is significant because the court finds that scraping public data did not violate the CFAA, though it violated LinkedIn's tos. LinkedIn ultimately wins at the end because hiQ was bankrupted. One of my take away is to scrape responsibly by rate limiting the requests and not overloading the server, etc
The goal of the companies and recruiters is to fill open positions with qualified candidate asap, hence why some would even pay job boards to promote their jobs or offer referral bonus. So my take is that companies would love job boards like this to freely advertise their positions to folks who might not otherwise come across these positions and increase their top of funnels. So in this case, the incentive of aggregators and companies are aligned and it is a win-win situation for both.
I don't know why a company might not want to partake in this, but would be happy to take it down upon their request if they like to make it more difficult for candidates to stump upon their openings.