That's an implementation choice. The standard makes no such guarantees.

> An integer may be converted to any pointer type. Except as previously specified, the result is implementation-defined, may not be correctly aligned, may not point to an entity of the referenced type, and can produce an indeterminate representation when stored into an object

The compiler may specify otherwise. GCC specifies that the pointer cast to integer back to pointer must point to the original object, else the behavior is undefined.