I am currently in the process of starting a project with Flask, SQLAlchemy, Celery. Say more about why I should avoid Celery and what to use instead.
I am currently in the process of starting a project with Flask, SQLAlchemy, Celery. Say more about why I should avoid Celery and what to use instead.
Things like chaining, groups, named queues just don't work the way you'd think they would. There's a lot of footguns and things require weird workarounds. Error reporting is misleading.
It's not bad enough where I had to pull it from the old project that used it, but going forward the new ones used a vibecoded queueing system that was genuinely more reliable than Celery but consumed a lot of memory (RSS inflation). Have then shifted to rq and at least for now it seems to "just work". You're better off doing anything custom/complex (like dependencies, or progress updates across multiple tasks) directly yourself in Redis anyway; since half the time Celery's less-well-trodden inbuilt features don't work the way they should anyway.
Huh that's interesting! I found celery to mostly match my expectations. I used it in a couple of django apps. My only real foot gun was around having to set an EAGER setting for local development or tasks never got executed.
How did you find your expectations and celery's actual semantics to be different? I'm trying to document well and it seems like I might have some implicit assumptions that I could make explicit, but I don't know what they are since they're already in my head and matching celery it seems.
What are you using Celery for? Do you need to be able to recover from a reboot or crash with your queues intact? Is it a distributed system as opposed to a single machine? Do you have complex multi-step workflows?
If you answered no to each question, just `import Queue from queue`.
> What are you using Celery for?
Things like provisioning, deploying, and eventually destroying cloud instances (VMs) on-demand when a user buys a specific service.
> Do you need to be able to recover from a reboot or crash with your queues intact?
Yes, I expect the queue to be durable.
> Is it a distributed system as opposed to a single machine?
Currently, everything runs on a single machine. But I expect it will eventually have to be split up. Although I do not expect it to be massively distributed or very complex.
> Do you have complex multi-step workflows?
Depends on what you mean by complex but Multi-step, yes.