Making it unsafe does nothing to prevent the semver issues. unsafe is used to deal with memory safety problems, it has no bearing on semver concerns, and it is not in any way useful to deal with semver concerns.
The cure you are proposing is a lot worse than the disease you want to fight with it.
We should probably add this proposal to Frequently Requested Changes - The Rust Language Design Team; it fundamentally clashes with core values of Rust, so it is extremely unlikely to ever happen.