Copy/paste from a previous thread [0]:
We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy in comparison to our existing use of MinIO, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s, over 10 nodes)
I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case.
---
In addition:
Next time we come to this we are going to look at RustFS [1], as well as Ceph/Rook [2].
We can see we're going to have to move away from MinIO in the foreseeable future. My hope is that the alternatives get a boost of interest given the direction MinIO is now taking.
[0]: https://news.ycombinator.com/item?id=46140342
[1]: https://rustfs.com/
[2]: https://rook.io/
Somewhat unrelated, but I just looked at the RustFS docs intro[1] after seeing it here. It has this statement:
> RustFS is a high-performance, distributed object storage software developed using Rust, the world's most popular memory-safe language.
I’m actually something of a Rust booster, and have used it professionally more than once (including working on a primarily Rust codebase for a while). But it’s hard to take a project’s docs seriously when it describes Rust as “the world’s most popular memory-safe language”. Java, JavaScript, Python, even C# - these all blow it out of the water in popularity and are unambiguously memory safe. I’ve had a lot more segfaults in Rust dependencies than I have in Java dependencies (though both are minuscule in comparison to e.g. C++ dependencies).
[1]: https://docs.rustfs.com/installation/
I agree that it is a bad idea to describe rust this way but they likely meant memory safety as used in https://www.ralfj.de/blog/2025/07/24/memory-safety.html . Meaning that shared mutable is thread unsafe, I am unsure about Java and JavaScript but I think that almost every language on the popular memory safe list fails this test.
Again the statement is probably still untrue and bad marketing, but I suspect this kind of reasoning was behind it
Of course Rust technically fails too since `unsafe` is a language feature
The docs of it and the marketing is a bit of a mess, tho I'm just gonna blame that on culture barrier as the devs are chinese
>Secure: RustFS is written in Rust, a memory-safe language, so it is 100% secure
[0]
qed
[0] https://docs.rustfs.com/installation/
Oh wow, they really wrote that. I love Rust but this is clown shit.
It's hard to take a project seriously if it focuses so much on the language it's written in. As a user, I don't care. Show me the results (bug tracker with low rate of issues), that's what I care about. Whether you program in Rust or C or Java or assembly or PHP.
As a potential user of an open source project, I care a fair bit what language it is implemented in. As an open source project, I preffer projects in languages and ecosystems I am familair and comfortable with. I may need to fix bugs, add features, or otherwise make contributions back to the project, and thus I am more likely to pick a solution in a language I am comfortable with than in a language I am not as comfortable with, given my other needs and priorities are met.
I agree, although I’m guessing they’re measuring “most popular” as in “most beloved” and not as in “most used.” That’s the metric that StackOverflow puts out each year.
I feel dumbfounded. All I've ever heard from rust users, is the equivalent of football fans running up, waving pendants in my face and screaming. So much so, that everything else said seems like the wild fantasies of "our team gonna win".
Then things like this appear:
https://www.phoronix.com/news/First-Linux-Rust-CVE
And I'm all warm and feeling schadenfreude.
To hear "yes, it's safer" and yet not "everyone on the planet not using rust is a moron!!!", is a nice change.
Frankly, the whole cargo side of rust has the same issues that node has, and that's silly beyond comprehension. Memory safe is almost a non-concern, compared to installing random, unvetted stuff. Cargo vet seems barely helpful here.
I'd want any language caring about security and code safety, to have a human audit every single diff, on every single package, and host those specific crates on locked down servers.
No, I don't care about "but that will slow down development and change!". Security needs to be first and front.
And until the Rust community addresses this, and its requirement for 234234 packages, it's a toy.
And yes, it can be done. And no, it doesn't require money. Debian's been doing just this very thing for decades, on a far, far, far larger scale. Debian developers gatekeep. They package. They test and take bug reports on specific packages. This is a solved problem.
Caring about 'memory safe!' is grand, but ignoring the rest of the ecosystem is absurd.
Debian has been doing this for decades, yes, but it is largely a volunteer effort, and it's become a meme how slow Debian is to release things.
I've long desired this approach (backporting security fixes) to be commercialized instead of the always-up-to-date-even-if-incompatible push, and on top of Red Hat, Suse, Canonical (with LTS), nobody has been doing it for product teams until recently (Chainguard seems to be doing this).
But, if you ignore speed, you also fail: others will build less secure products and conquer the market, and your product has no future.
The real engineering trick is to be fast and build new things, which is why we need supply chain commoditized stewards (for a fee) that will solve this problem for you and others at scale!
> Debian has been doing this for decades, yes, but it is largely a volunteer effort, and it's become a meme how slow Debian is to release things.
which is a bit silly considering that if you want fast, most packages land in testing/unstable pretty quickly.
But then you as a consumer/user of Debian packages need to stay on top of things when they change in backwards-incompatible ways.
I believe the sweet spot is Debian-like stable as the base platform to build on top of, and then commercial-support in a similar way for any dependencies you must have more recent versions on top.
> But then you as a consumer/user of Debian packages need to stay on top of things when they change in backwards-incompatible ways.
If you need latest packages, you have to do it anyway.
> I believe the sweet spot is Debian-like stable as the base platform to build on top of, and then commercial-support in a similar way for any dependencies you must have more recent versions on top.
That if the company can build packages properly. Also too old OS deps sometimes do throw wrench in the works.
Tho frankly "latest Debian Testing" have far smaller chance breaking something than "latest piece of software that couldn't figure out how to upstream to Debian"
The difference is between staying on stable and cherry-picking the latest for what you really do need, and being on everything latest.
The latter has a huge maintenance burden, the former is the, as I said already, sweet spot. (And let's not talk about combining stable/testing, any machine I tried that on got into an non-upgradeable mess quickly)
I am not saying it is easy, which is exactly why I think it should be a commercial service that you pay for for it to actually survive.
I would love to see a software distribution model in which we could pay for vetted libraries, from bodies that we trust, which would become FOSS after a time period - even a month would be fine.
There are flaws in my argument, but it is a safer option than the current normal practices.
When it is tailored to one customer, that dependency being maintained for you is probably a very particular version you care about. So while copylefted code you can always reshare, it's the timeliness and binary package archives that are where the value really is.
Not dismissing your point, but Looking at the article, it looks like it's in rust unsafe code. Which seems to me to be a point that the rest of the rust code is fine but the place where they turned off the static safety the language provides they got bit.
Hey! Can't I just enjoy my schadenfreude in peace?
I guess the takeaway is that, doubly so, trusting rust code to be memory safe, simply because it is rust isn't sensible. All its protections can simple be invalidated, and an end user would never know.
Um I doubt Debian maintainers look at every single line of code in the packages they maintain.
One might even call the rust community a “cargo cult”
They explicitly say that top performance is not a goal: «high performances constrain a lot the design and the infrastructure; we seek performances through minimalism only» (https://garagehq.deuxfleurs.fr/documentation/design/goals/)
But it might be interesting to see where the time is spent. I suspect they may be doing fewer things in parallel than MinIO, but maybe it's something entirely different.
> We can see we're going to have to move away from MinIO in the foreseeable future.
My favorite thing about all of this is that I had just invested a ton of time in understanding MinIO and its Kubernetes operator and got everything into a state that I felt good about. I was nearly ready to deploy it to production when the announcement was released that they would not be supporting it.
I’m somewhat surprised that no one is forking it (or I haven’t heard about any organizations of consequence stepping up anyway) instead of all of these projects to rebuild it from scratch.
I wouldn’t use rook if you solely want S3. It is a massively complex system which you really need to invest in understanding or else your cluster will croak at some point and you will have no idea on how to fix it.
IS there a better solution for self-healing S3 storage that you could recommend? I'm also curious what will make a rook cluster croak after some time and what kind of maintenance is required in your experience.
I have unfortunately got a ceph cluster in a bad enough state that I just had to delete the pools and start from scratch. It was due to improper sequencing when removing OSDs, but that is kindof the point is you have to know what you are doing to know how to do things safely. For the most part I have so far learned by blundering things and learning hard lessons. Ceph clusters when mistreated can get into death spirals that only an experienced practitioner can advert through very carefully modifying cluster state through things like upmaps. You also need to make sure you understand your failure domains and how to spread mons and osds across the domains to properly handle failure. Lots of people don’t think about this and then one day a rack goes poof and you didn’t replicate your data across racks and you have data loss. Same thing with mons, you should be deploying mons across at least 3 failure domains (ideally 3 different datacenters) to maintain quorum during an outage.
Not used it yet, but RustFS sounds like it has self healing
https://docs.rustfs.com/troubleshooting/healing.html
ceph?
Rook is ceph.
My small adventure with rustfs is that it is somewhat underbaked at the moment.
And also it is already rigged for a rug-pull
https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens...
yeah, their docs look pretty comprehensive, but there's a disturbing number of 404s that scream "not ready for prime-time" to me.
from https://rustfs.com/ if you click Documentation, it takes you to their main docs site. there's a nav header at the top, if you click Docs there...it 404s.
"Single Node Multiple Disk Installation" is a 404. ditto "Terminology Explanation". and "Troubleshooting > Node Failure". and "RustFS Performance Comparison".
on the 404 page, there's a "take me home" button...which also leads to a 404.
Please also consider including SeaweedFS in the testing.
Having just finished a "hobby size" setup of Rook-Ceph on 3 n100 mini pcs, with every service to fit in a couple hundred MB of ram (one service needs up to 3Gb when starting, but then runs around 250MB) I'd ask why not ceph?
At work I'm typically a consumer of such services from large cloud providers. I read in few places how "difficult" it is, how you need "4GB minimum RAM for most services" and how "friends do not let friends run Ceph below 10Gb".
But this setup runs on a non dedicated 2.5Gb interface (there is VLAN segmentation and careful QoSing).
My benchmarks show I'm primarily network latency and bandwidth limited. By the very definition you can't get better than that.
There were many factors why I chose Ceph and not Garage, Seaweed or MinIo. (One of the biggest is that ceph does 2 birds with one stone for me - block and object).
Ceph is far higher on RAM usage and complexity. Yeah if you need block storage in addition it's a good choice, but for anything smaller than half a rack of devices it's kinda overkill
Also from our experience the docs outright lie about ceph's OSD memory usage and we've seen double or more than what docs claim (8-10GB instead of 4)
>I get the impression that Garage isn’t necessarily focussed on this kind of use case.
I wouldn't be surprised if this will be fixed sometime in the future.
Those rates are peanuts considering that a decade ago saturating 40G, per core, was more than reasonable via standard userspace networking, with atleast a few copies in the datapath.
passing blocks of memory around vs referencing filesystem/database, ACLs, authentication and SSL