This can be done in an unambiguous way. For example if the appropriate calls are added to the primitive types the annotation only affects which call is invoked when using operators. (IE the meaning of + or * is changed in that scope)
This of course means the answer to "Does this affect .sum(), .sqrt() etc." is "No".
On the surface it might seem like the lack of propagation to called methods is a downside. However I think it may be necessary regardless of the approach.
It ought to be possible for a particular crate / method to control its own meaning.
It is highly undesirable for an application to depend on a library, where the library has many tests to check its correctness, but these are circumvented because the application it is embedded into changes the meaning of multiplication inside of it. This pitfall is precisely why --fast-math has a bad reputation.
Regardless of syntax, if the control is local, we can maintain consistency and testability and don't need to regard it as unsafe.