We upgraded to 2.6.1 about a week ago and switched to using the new(ish) parallel(ish) garbage collector. I still can't tell what the impact has been.
Claude Code (which is a wizard at analyzing log files but also, I fear, an incorrigible curve-fitter) insisted that it was a real breakthrough and an excellent choice! On the other hand there was a major slowdown last night, ending in SBCL dying from heap exhaustion. I haven't had a chance to dig into that yet.
If by the cap you mean the heap size passed in as the --dynamic-space-size argument, then it definitely didn't hit the cap. It was using about 2/3 of that.
> My understanding is the new parallel GC compacts the heap rather infrequently
I'm going to caveat this by stating up front that obviously HN's source code is not public so I don't know what your hot path looks like, and that I'm not a domain expert on garbage collection, but I do write a fair amount of lisp for SBCL.
Immix-style collectors, like the new GC in SBCL, only compact on an opportunistic basis and so you get fragmentation pressure under load. In that situation, you might be well under the dynamic space size cap but if it can't find a large enough contiguous chunk of free heap it will still die.
So, fragmentation would be my prime suspect given what you described.
We upgraded to 2.6.1 about a week ago and switched to using the new(ish) parallel(ish) garbage collector. I still can't tell what the impact has been.
Claude Code (which is a wizard at analyzing log files but also, I fear, an incorrigible curve-fitter) insisted that it was a real breakthrough and an excellent choice! On the other hand there was a major slowdown last night, ending in SBCL dying from heap exhaustion. I haven't had a chance to dig into that yet.
>SBCL dying from heap exhaustion
Due to hitting the cap, or to framentation? My understanding is the new parallel GC compacts the heap rather infrequently.
If by the cap you mean the heap size passed in as the --dynamic-space-size argument, then it definitely didn't hit the cap. It was using about 2/3 of that.
> My understanding is the new parallel GC compacts the heap rather infrequently
Can you explain more?
I'm going to caveat this by stating up front that obviously HN's source code is not public so I don't know what your hot path looks like, and that I'm not a domain expert on garbage collection, but I do write a fair amount of lisp for SBCL.
Immix-style collectors, like the new GC in SBCL, only compact on an opportunistic basis and so you get fragmentation pressure under load. In that situation, you might be well under the dynamic space size cap but if it can't find a large enough contiguous chunk of free heap it will still die.
So, fragmentation would be my prime suspect given what you described.
Sorry for suddenly clinging to you for support but might we be better off using the older GC in that case?
Thanks. Your link gives more insight into "why submit now?" Appreciate it.