I don't see it mentioned enough in the comments here, but not considering Celery as an alternative to Django + async really is the missing puzzle piece here. Aside from application-level options that weren't explored, I'm wondering whether handling some of the file IO stuff with, for instance, nginx, might be a better fit for their use case.
Once you're in the situation of supporting a production system with some of the limitations mentioned, you also owe it to yourself to truly evaluate all available options. A rewrite is rarely the right solution. From an engineering standpoint, assuming you knew the requirements pretty early on, painting yourself into a bad enough corner to scrap the whole thing and pick a new language gives me significant pause for thought.
In all honesty I consider a lot of this blog post to be a real cause for concern -- the tone, the conflating arguments (if your tests were bad before, just revisit them), the premature concern around scaling. It really feels like they may have jumped to an expensive conclusion without adequate research.
In an interview, I would not advance a candidate like this. If I had a report who exhibited this kind of reasoning, I'd be drilling them on fundamentals and double-checking their work through the entire engineering process.
I tried to use Celery for something extremely trivial (granted, 5+ years ago). It was so badly documented and failed to do basic things I would expect from a task queue (like progress reporting) I have no idea why it was and still is popular.
Just because you couldn't figure it out doesn't mean the capability wasn't there. More than ten years ago at this point I was running a massively scaled Celery + RabbitMQ + Redis deployment with excellent off-the-shelf reporting using Flower.
I'm going to back GP, eight years ago my team developed a system that was basically an async, scheduled, async task queue, we used celery, and it was so buggy and so troublesome that at some point we switched to a hastily hacked own solution and it worked better.
It's entirely likely that we did something wrong and misused celery. But if many people have problems with using a system correctly then it's also something worth considering.
I hated celery so much I wrote my own message queue.
There’s not much software I really dislike but Celery is one.
A nightmare within a nightmare to configure and run.
Hi! Could you please tell me what use cases would nginx be better for, outside of serving static files?
Moreover, having worked with Django a bit (I certainly don't have as much experience as you do), it seems to me that anything that benefits from asynchrony and is trivial in Node is indeed a pain in Django. Good observability is much harder to achieve (tools generally support Node and its asynchrony out of the box, async python not so much), Celery is decent for long running, background, or fire and forget tasks, but e.g. using it to do some quick parallel work, that'd be a simple Promise.all() is much less performant (serialize your args, put it in redis, wait for a worker to pick it up, etc), doing anything that blocks a thread for a little bit, whether in Django or Celery,is a problem, because you've got a very finite amount of threads (unless you use gevent, which patches stdlib, which is a huge smell in itself), and it's easy to run out of them... Sure, you can work around anything, but with Node you don't have to think about any of this, it just works.
When you're still small, isn't taking a week to move to Node a better choice than first evaluating a solution to each problem, implementing solutions, each of which can be more or less smelly (which is something each of your engs will have to learn and maintain... We use celery for this, nginx for that, also gevent here because yada yada, etc etc), which in total might take more days and put a much bigger strain on you in the long term? Whereas with Node, you spend a week, and it all just works in a standard way that everyone understands. It seems to me that exploring other options first would indeed be a better choice, but for a bigger project, not when the rewrite is that small.
Thank you for your answers!
[dead]