The reason pointers are UB is that they could be anything including another object, code or memory mapped hardware interfaces. The analog here would be if you would just use the same index into a different arena, that trouble also wouldn't be bounded to just one arena.