Actually I am not such a big fan of this t-shirt, it makes UB much more "mysterious" than it has to be. Also see this talk.
(Well, truth be told I like the shirt but not the message. 
)
Basically, there is a contract between you and the compiler when writing code in a language (not different from the contract you might have when using a library API or a network protocol), and UB is just a different word for a contract violation.
I don't think this is correct. Rust without Rc and mem::forget actually does not leak memory. Well, there are probably a bunch of containers that are not panic safe in this model, i.e. they skip destructors on panic, but that can be fixed. In fact let's just drop all containers except for Vec, and lets reduce that to the core API (e.g., no drain, which I believe can leak on panics). This is a leak-free language. And then we can implement a Turing machine in that language to show it is Turing complete.
Arguing with the halting problem makes no sense. Deciding whether a program dereferences a NULL pointer is also equivalent to the halting problem and that doesn't stop us from preventing this with static checks. We are just willing to also have rustc reject some correct programs. Similarly, our type system does not have to exactly characterize leak-free programs.