The problem is that specialization causes action at a distance. If you have an unsafe specialization trait and have it implemented conditionally on a safe trait then your safe trait becomes a source of unsoundness if unsafe code somewhere depends on the distinction. The lifetime-dependence is usually tricky to weaponize, but when it comes to unsoundness the bar is higher than that.
A subset of specialization could be stabilized. But that would lock us into the current syntax which might not be sufficient for lattice specialization.
Related thread: Idea: making traits specialization safe over an edition boundary