That is not what any compiler writer I know would say, for all compilers I'm aware of assume that casting a pointer to an integer causes the pointer to escape and every integer-to-pointer conversion creates a pointer that may alias any external pointer (including the aforementioned escaping pointers). The problem that's been identified is that this is not sufficient.
If you were to describe the semantics operationally, you'd observe that pointers carry information in them that are more than just the integer address of its location, and this extra information is not saved upon conversion to integer in a naïve semantics. Converting an integer back into a pointer must generate some information for the pointer. A naïve semantics would imply that a pointer-to-integer-to-pointer roundtrip is equivalent to doing nothing at all (on the basis of it all being nops at hardware level), but careful exploration of operational semantics suggests that it is equivalent to clearing that extra information fields, and it is not a valid refinement to fill in that information with a subset of everything.
With LLVM, there is another wrinkle, in that memory is untyped, so effectively every load or store instruction is converting its type to an integer. So LLVM is permitted (and used to until recently, albeit very controversially!) to turn a store (load i8*) into store (load i64), which will introduce inttoptr/ptrtoint into code where the user previously only pointers. (And yes, this has caused optimization problems--I've been bitten by this "bug"). Take the implicit inttoptr literally, and literally every pointer variable in your program escapes because of the presence of the memory inttoptr; ignore the implicit inttoptr entirely and you probably introduce a soundness hole by folding a user inttoptr into the implicit memory one and miss that you actually need one.
Semantics are difficult, even for the people who know what they're doing. A lot of potential pitfalls with semantics aren't apparent until you can find examples of where things go wrong or try to write the optimizations that you intend to write. C++'s experience with relaxed atomics is an example of where something seemingly simple turned out to be more complex than expected, and the issues LLVM has with inttoptr and undefined values are a long-running exercise in "vague, obvious semantics don't work, and most compiler authors aren't prepared to reason about these in a more correct semantic model" (it doesn't help that the LLVM community isn't particularly welcoming towards formalizing their semantic model).