That's fair. However, unfortunately, the only way I know to make that true is to actually use a simple, naive, non-optimizing compiler. To realize the performance goals Rust is aspiring to hit, we need optimizations, and many of the interesting ones are inherently incompatible with a naive model.
“pointer type casts do nothing, because they’re all just the same integer”
Note that pointers are far from being just integers even if you ignore Stacked Borrows, and even in C.