Another problem you haven't mentioned is Drop. You'd effectively make these types (pseudo) linear, as you need to provide the real allocator handle in order to drop it.
This is somewhat mitigated by "always" having the real allocator stored in a wrapping type whose Drop impl does a real drop for its children, but it's still problematic during construction and in general.
See also Is custom allocators the right abstraction? - #6 by matklad. There's some experimentation with callsite injected storages.
Prior art is the raw_entry API for HashMap which allows providing hash/eq implementations at call time, but general feelings seem to be that doing similar for allocators is much more error prone (and unsafe the whole way down, unlike HashMap's raw_entry).