> Sadly, solutions like BetterTouchTool and InstantSpaceSwitcher won't work for me because I prefer to use my trackpad to switch spaces.
One of BetterTouchTool's first features ~17 years ago was trackpad gesture customization, it is still one of the most important things you can do with BTT! ;-) You'd just need to assign the "Move Right a Space (without animation)" and "Move Left a Space (without animation)" actions to trackpad gestures in BTT.
I don't want it "without animation", I like that the animation tracks my fingers and that the response is instant and doesn't wait until a "gesture" is "triggered". I just want it to wait a second after I let go until the target workspace starts receiving input.