Formal specs and optimizations in general

You're right. I actually came to a similar conclusion earlier after talking with @RalfJung:

Yeah, I actually feel like the pointers question is a just a symptom of the problem. The primary cause is that without a formal definition, it's hard to do formal logic. Floating point is also a hard problem. Memory is a hard problem (different levels of caches with different accessibility, memory mapped I/O, memory mapping different address spaces together, etc.) In short, there are an endless series of hard problems, with pointers being just one small facet of the problem.