Yes, that would work. But you would need to carry pointer to allocator inside box and it is extra code to run for every object.

extra code meaning the drop implementation or something else?