Can you help an old man understand?
This sounds like a kafka-like streaming system, but backed onto s3-objects?
Doesn't this mean that write performance is going to be bad?
Can you help an old man understand?
This sounds like a kafka-like streaming system, but backed onto s3-objects?
Doesn't this mean that write performance is going to be bad?
I'm glad you asked, and yes, PicoMQ does have some Kafka-like semantics. However, Kafka is great at being a huge pipe, so you'd create topics like tables. PicoMQ, on the other hand, recommends creating granular streams that make the most sense, say, by user, session, or vehicle (still bottomless).
And it's also fair to question write performance, since it's backed by object storage. The optimization is primarily from the shared WAL across streams, server-side batching, and client-side in-memory pipelining, especially with HTTP/2, without as much connection pool overhead.
In practice, you can go to the extent of achieving up to 100 MiB/s throughput per stream. Considering how granular streams can be, you'd rarely need as much. The latency for a durability ACK is, however, the price to pay, which is going to be ~250 ms, or lower with S3 Express, which I'd say covers most real-time use-cases. The design itself is easy enough to extend to a disk-staged WAL for single-digit durability ACK latency.
Have you look at Google's new Rapid Bucket offering? Google Cloud Storage is arguably as good as S3, and is protocol compatible; Rapid Buckets are a type of bucket which supports appendable objects and low latency I/O. The downside is they can only be zonal, and they're a bit more expensive.
PicoMQ works with any S3-compatible object store. But I wasn't aware of GCS Rapid Bucket, it sounds a lot like AWS S3 Express, which is also zonal. And it does help with durability ACK latency quite a bit, keeping it closer to ~50ms.
I'll be setting up a GCP deployment example similar to AWS soon. I'll be sure to try Rapid Bucket as well, thanks for sharing!
Not familiar with this particular library, but similar libraries use S3 Express One Zone which has write latency <10ms, so you can use that for the WAL and compaction can move data onto other storage classes in the background.
Regular S3 has write latency 100-150ms, which might be fine depending on your workload anyway.
You nailed it! With some of the similar products I've seen, they either inherit the Kafka protocol and hence the KRaft and other complexities, or go the other way with single-node only deployments, commonly just using SlateDB's single-writer model for durability.