I suspect this doesn't work in practice. My impression is that a very large portion of unsafe code is either for FFI (where turning it off isn't even an option) or for optimization (which you would typically want on by default).
In particular, there's a very strong risk of any audit framework/tooling like this unintentionally leading to crate authors being discouraged from using any unsafe code for optimization, even when the soundness of that unsafe code is uncontroversial. In past discussions this concern has sometimes been expressed as "demonising unsafe code". Unfortunately I have no good ideas on how to prevent this, but I do think that we'd be causing more harm than good if we introduced a system that did have this problem in practice (if nothing else, it risks encouraging the idea that security is at odds with performance).
Pedantic but important: We'd want to check the entire module for changes, for every module containing at least one unsafe.