Setting aside whether or not you can pull off a lock free approach here we can be certain of a couple things. There will be at least some overhead that must be paid somewhere even if that's on a separate management thread. And there will be a lot of additional complexity because that's just how concurrency always is.

Meanwhile the better the scheduler performs the more competitive the thread local approach becomes.