It all comes down to whether pointers into the arena do something different than normal pointers when they are dangling.
Normal pointers cause UB. That’s astronomically bad. If your arena pointers crash your program in a well-defined way, that’s already a much better situation.
The reason pointers are UB is that they could be anything including another object, code or memory mapped hardware interfaces. The analog here would be if you would just use the same index into a different arena, that trouble also wouldn't be bounded to just one arena.