I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one.
mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in multithreaded scenarios (and only just 8 threads). Switching to mimalloc improved performance by 20x, very close to what glibc offers by default, and just a bit under a glibc + mimalloc configuration.
I get that there's a real issue there and it's interesting (to some) to address it, but it should have never surfaced this way in the first place.
> I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient).
ripgrep actually sets jemalloc as global allocator when built for 64b musl: https://github.com/BurntSushi/ripgrep/blob/435f59fc4b43af3ab...
That's good! But I wonder why it wasn't then enabled for that configuration or why the override wasn't "global" enough and the default allocator was still partly used.
Rust doesn't get to override musl internals. Ripgrep uses opendir, a POSIX library feature implemented in musl to look at er, directories. The stack trace suggests we blew up when Rust's std::sys::fs::unix::readdir internal detail called opendir, and it in turn allocated.
On Linux it would be possible for ripgrep to talk directly to the kernel via documented system calls without libc, but that wouldn't work on any other popular OS.
> Rust doesn't get to override musl internals.
Is malloc not a weak symbol in musl? I would expect it to be overridable like it is with glibc. Or does ripgrep only override Rust's global allocator?
> Or does ripgrep only override Rust's global allocator?
That one, as you can see in the code linked above, it only does rust-level overriding as that was where musl's allocator was found to impact ripgrep (https://github.com/BurntSushi/ripgrep/commit/03bf37ff4a29361...)
Last time I checked (a while ago), it wasn't. You needed to compile musl and exclude a few files from the archive to then provide your own allocator.
Depending on the toolchain, I'm assuming you could use some tricks or hacks to make it work better, but the tooling I used (Bazel) did that automatically and I haven't bothered to look at the internals yet .
I assume it already doesn't work on Windows. At some point you have to define your compatibility boundary. And high performance often coincides with mediocre compatibility.
You can call into the kernel just fine in Windows. The fact you use a function call wrapper instead of a raw syscall is not really relevant.
Most Windows NT kernel functions take a buffer to use rather than allocating their own memory. Even many Win32 functions don't allocate (although there you have to be much more careful).
A good example is go. On linux you can use a from scratch image fairly easily because it only uses syscalls. But for windows or mac the moving target wasn't maintainable so they link against shared objects. Linus enforcing the don't break userspace rule is what made that possible. That definitely has tradeoffs. At some point relibc or something similar will allow the same (stably) for rust. But using posix as that compatibility boundary gets you a much larger set of OS and only occasionally has a performance penalty. Often, a posix api tuned to the kernel is more performant. Musl is an exception precisely because it makes an openbsd-esque trade of performance for simple small attack surface.
The Linux kernel defines syscalls as its stable ABI (note: there are also non-kernel ABIs on Linux, such as Wayland) while Windows defines the DLL calls as its stable ABI.
One isn't better than the other. Linux's approach allows binaries to be fully statically linked, which is a more predictable environment for binaries, but Windows's approach composes better, as every process loads DLLs and this allows for things like graphics drivers and COM to work more reliably. As things stand on Linux you can't use the GPU in a portable statically linked app, because the kernel doesn't define the semantics of dynamic linking.
FYI, Windows's approach is better in some respects, like letting you avoid syscall overhead in some cases.
Notably, Linux has this too, but it's a special mechanism (vDSO) while on Windows it's just a normal function call with no difference from any other function.
If you look at the sigsegv stack, the allocation comes from opendir which is in musl libc as well. The allocation override mechanism used in Rust doesn’t replace the allocator process-wide; it merely replaces the allocator Rust code talks to.
It does seem like ripgrep should probably avoid using opendir from libc if it allocates using an allocator with a global lock though.
It’s a kernel bug. While I agree libc allocators suck for no good reason, it seems like this work of equally likely hit other application code including mimalloc and glibc.
Sure, but the problem isn't that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator.
mallocng should not be used at all.
Ah, no, the problem is that the bug exists.
Another, separate, problem, noticed by people investigating this problem, is that ripgrep on musl uses a serializing allocator in the hot path.
it's a tradeoff; mallocng does have benefits as well -- for example it uses less than half the amount of RAM compared to mimalloc and glibc for certain Python workloads. While yes it is a bit slower, I prefer mallocng in many cases.
I imagine there is no free lunch in the allocator space, but a series of trade-offs. Pick your poison on implementation which is going to be sub-optimal for some subset of use cases.
Without a hugely compelling reason to switch, going with the default is reasonable.
ripgrep is there to be FAST. Trading any speed to improve memory efficiency over a longer period of time for a process with a short life-time doesn't make sense in this case.
It's also a development tool. If your development machine is having RAM issues because it's doing a grep, you have bigger problems to solve.
As for other workloads that might use less RAM with mallocng compared to other performant ones, I'm curious to know about the magnitudes we're talking about. From 10MB to 20MB or from 100MB to 2GB? How was the speed of the program? Was there any multithreading involved?
For context, the Python workload was a general-purpose fileserver with file indexing and image thumbnailing, largely IO-bound. Switching from mallocng to mimalloc resulted in a slight speedup (150% of baseline), but over twice the memory usage, going from 250 to 670 MiB, primarily for thumbnailing. Some pathological cases (libvips calling imagemagick to decode heif) was a 10x multiplier.
There was multithreading, and yes, mallocng visibly became a bottleneck beyond 5 threads. However already at 3 threads there was diminishing returns for both allocators, so this was not an issue in this case.
The speed gain was hardly noticeable in practice since the program was already plenty fast, but the additional memory usage was a very real inconvenience.
Is that memory increase seen in peak usage or average/idle usage?
For file servers, I could be willing to give up 350 MB of memory temporarily for large-scale indexing and thumbnailing operations if it doesn't happen often and makes the file serving/browsing more responsive.
On the other hand, plenty of NAS/file server systems are going to have paltry CPU and memory. Running slower with a more modest memory foot print might be the only way to keep the system stable.
libvips shouldn't be calling imagemagick for heif decode, it has a nice one built in. Unless you were using a very old libvips!
I've found jemalloc works best for long running libvips processes, fwiw.
It's maybe a kernel bug.
The way most programs achieve being fast is by re-using allocations. You don't need a fast allocator if you don't allocate. Nothing of what ripgrep does inherently requires frequent allocations.
The ISO C definition of opendir() requires an allocation in practice because it returns a DIR* and it's bad practice for the library to arbitrarily limit how many of those an application has at a time.
Maybe a C library could preallocate several DIRs and only use the heap when those are exhausted, but this ripgrep use case (lots of threads running in parallel on a large tree) would still be likely to trigger that.
There is no ISO C definition of opendir(), that's a POSIX thing.
But yeah, those DIR objects which opendir() returns a pointer to are probably some kind of heap allocated. But we're talking about a system call involving the filesystem here. The time taken by the allocator is gonna be dwarfed by the time spent in the syscall even with the slowest allocator.
Ripgrep reads through every byte of most files and matches it against a regex. That is the tight inner loop where you want to avoid allocations.
Not that you even need to call the C functions. I don't think ripgrep would gain anything from it, but the syscall to read directory contents just needs a file descriptor.
I think ripgrep does avoid allocations in that innermost loop. That's why this report says all the crashes are associated with opendir() and its memory allocation: the opendir() call is outside of ripgrep's innermost loop but still runs often enough to trigger the race condition.
And that's why musl's allocator's performance isn't a huge concern for ripgrep
That's to be properly measured. Assuming that the allocator will not cause issues there is to be proven.
The application I recently improved by changing the allocator had a similar profile (a C++ include scanner) and thread parallel I/O functions had terrible performance originally with mallocng. Adding threads almost had negative value because of the contention.
We use musl+mimalloc by default for our entire production operating systems: https://stagex.tools
There is a real tradeoff:
- The musl allocator is only slow with multi-threading.
- Almost all other allocators have trouble reclaiming memory when using multi-threading. This often results in multiples more RSS than single threaded or musl's allocator.
Agree with you on mimalloc. It can even be configured to be aggressive in memory reclaim at the cost of performance.