For the purpose of blurring, there's nothing wrong with linear RGB.
HSL has the angle discontinuity which is really annoying to deal with and sometimes even impossible to work around.
CIELAB should be fine in theory, but it's not what people actually use on their screens, so I've had some problems with it in practice.
CIEXYZ (CIE 1931) had the best brightness axis ever. But the hunt for the best color-axis seems to be ongoing. CIE 1960, CIE 1964, CIELAB/CIELUV 1976...
CIECAM02 CIECAM16 seem to be new iterations. Maybe those are good now?