Unless I misunderstood your post, you're creating references to uninitialized types which may be UB (depending on the final the rules) and is definitely a bit dangerous (you're producing safe references to types with "safe" methods that are actually unsafe to use). Isn't that why you initially started out by using *mut T pointers everywhere? The fact that you had to make this choice leads me to believe that there's something missing so I'm trying to propose a less "unsafe" middle ground ("unsafe" code doesn't have to be all or nothing).
After thinking about it, you don't actually need a special vector to deal with this; a simple Uninit<T> wrapper and a Vec<Uninit<T>> should suffice. I've forked off a new topic as it doesn't really seem to belong here: