You could definitely do better than x32 did (IIRC it is a bit of an outlier even among "32-bit compat ABI" setups). But even if the kernel changes were done more cleanly that still leaves the whole software stack with the ongoing maintenance burden. The fact that approximately nobody has taken up x32 suggests that the performance gain is not worth it in practice for most people and codebases.
Defining yet another 32-bit-on-64-bit x86 ABI would be even worse, because now everybody would have to support x32 for the niche users who are still using that, plus your new 32-bit ABI as well.
But that maintenance burden has been paid off for things like 64-bit time_t on 32-bit ABI's. One couod argue that this changes the calculus of whether it's worth it to deprecate the old x32 (as has been proposed already) but also propose more general "ABI-like" ways of letting a process only deal with a limited range of virtual address space, be that 32-bit, 48-bit or whatever - which is, arguably, where most of the gain in "x32" is.