Not what you're after, but your talk of beans reminded me of Taco Bell Programming. [0]
[0] https://widgetsandshit.com/teddziuba/2010/10/taco-bell-progr...
Not what you're after, but your talk of beans reminded me of Taco Bell Programming. [0]
[0] https://widgetsandshit.com/teddziuba/2010/10/taco-bell-progr...
His example of a web crawler stinks. I can get behind the philosophy, but his specific web crawler example at a minimum needs to retry, respect robots.txt, and rate limit per domain. So no, xargs + curl is a dumb example.
Replace curl with wget, get robots.txt support. Write timed-out URLs into a file for the next identical run.
The idea stays: you can jerry-rig a web crawler from pedestrian, readily available parts, and quickly test the idea for which you wanted to crawl the Web.
The point is that bash, xargs, wget, make, etc are well-understood, well-maintained tools. They are not even necessarily simple, but they likely contain very few unknown unknowns.
Based on bot traffic to my personal blog over the past year or so, the minimum viable web crawler apparently does not need any of those things (much to my own personal annoyance, to be very clear).
Nah. It’s fine.
Speaking of beans, here's a classic (totally different from the main topic. Welcome to Topic Creep):
https://archive.uie.com/brainsparks/2011/07/08/beans-and-nos...
This was a great read, thanks! I've naively tried to stop too many beans-and-noses situations in the past.