Pointers are complicated, and much more than a number. Pointers are both the actual (well, virtual) address of memory, as well as context about how that pointer is intended to be used: that is the thesis of the work @RalfJung has worked on with their “Aliased Borrows” model and MIRI. TL;DR we aren’t completely sure what the model is yet.
“isomorphic” in which sense?
Raw pointers have pretty much no guarantees attached to them. So you can do almost anything. But that’s not a useful statement, because all it means is that you have to be careful when accessing a raw pointer.