2 am is mid afternoon for the Pune office. If you’re running a 24/7 operation and all of your employees are in PST you’re gonna have a bad time.

But the other thing that has changed since 2013 is that most of us are using autoscaling. There were times of day when there is supplemental server capacity to run periodic processes, and it made sense to schedule them during quiet daylight hours if they existed, but now the availability of bus numbers if and when a problem happens are more pressing. And they all outweigh concerns about DST. Instead of seeking the low traffic times of day you mainly need to avoid peak traffic.

The batch processing system I renovated to use 7% if the CPU hours of the original design, used the same fixed parallelism trick that crawlers use so as not to knock over your sites. They keep k requests in flight at all times and if your server responds faster, they issue more requests per second. If the production services had excess capacity the job would run in ten minutes. If response times are slow it takes 15. You’re inverting Little’s Law and keeping the amount of servers fixed and varying the duration.

It’s a pity that spooling up more servers to handle that job took a substantial fraction of that ten minutes or I could have gotten it to five (originally it was over 30 minutes and climbing toward 35). I tried at one point to move it entirely to a Bamboo agent but we underprovisioned them. If I’d had an agent with 2x the CPUs I could have shut down a service. I was on my way out the door before I realized that upgrading the bamboo agents would have cost us less than running that service and everyone’s builds would have been faster as a consequence. And probably let someone else do the same. False economies.