On the same boat, and honestly, it’s not getting the attention it deserves. Given Databricks acquired ElectricSQL primarily for PGlite, the Durable Streams project is likely going to be abandoned unless new maintainers step in.
I did, in fact, initially write PicoMQ with OpenRaft, similar to Ursala, but I really wanted the operational complexity to be minimal and the nodes to be stateless (at least for most use cases), Pico uses SQL database as a metadata command log, inspired by RisingWave.
But Ursala would, without a doubt, have better durability ACK latency, as it wouldn’t have to wait for an ACK from S3. That said, I do plan on extending disk/EBS-staged WAL to hit similar low-latency durability ACK numbers. But again, most use cases don’t need single-digit-millisecond latency for ACKs.
I was planning to use electric cloud for durable streams recently and found out they are winding it down post acquisition.
Unfortunate timing and self hosting isn't complex or anything but still as you said, durable streams as a project is most likely going to be abandoned.
True, there are also a lot of open PRs and issues. I'm surprised no other developers or orgs are stepping in to keep the project afloat. No wonder the Apache Way goes a long way! Either way, their approach of building around an open-protocol was a good decision.
If you'd like to consider using PicoMQ, I have examples to deploy on Fly.io and AWS. ursula.tonbo.io is also a good option to try.
Thanks for the reply, sounds great, going to dive in!
Your welcome! And feel free to ask any questions as try out PicoMQ or raise on Github issues. The disk-staged WAL is currently tracked here: https://github.com/PicoMQ/picomq/issues/13, I'm planning to ship it as an optional add-on.