I feel like this comment summarizes it pretty well (slightly extended by me to also cover unsafe blocks):
Any time unsafe is on a Trait or a block the implementer must verify it for safety, any time unsafe is on a function or method (trait or inherent) the caller must verify it for safety at each use.
So, I agree that there are "two polarities" here, and there's no fundamental reason these should have the same keyword. IIRC there actually was a proposal once to change unsafe blocks into trusted {} (and then unsafe traits should require a trusted impl) to convey the idea that this block doesn't actually do anything bad -- it's just that we have to trust it to not do anything bad because the compiler can't check it. I also feel it may have been a mistake to permit the entire body of an unsafe fn to perform unsafe operations; just because we rely on additional invariants doesn't mean we want to accidentally deref a raw pointer somewhere.
I do not see how unsafe blocks can be seen as an effect. Effects are characterized operationally, things like "this function will not read from or write to the heap in any way". unsafe cannot be characterized that way. Quite the opposite, actually: One important use-case of unsafe blocks is to make some function or method go fast without changing behavior compared to the slower safe implementation (think of using get_unchecked_mut). The fact that this function uses unsafe is not observable from the outside at all, so it's not an effect.
The fact that unsafety is typically encapsulated where effects are "infectious" as you said, also indicates to me that viewing unsafe as an effect is not very useful. unsafe fn foo(x: u32) indicates a precondition not expressible in the type-system ("x must be even"). If a caller can guarantee that that precondition always holds (by calling foo(2*y)), the precondition is discharged and any unsafety taken care of and perfectly hidden from the sight of the caller. This is in strong contrast to the "outwards propagation" of effects.
I think of unsafe code as being essentially untyped: We have a language that allows all sorts of dangerous operations, and we have a type system that syntactically carves out a well-behaved subset of functions and data structures. However, the set of all well-behaved functions cannot be characterized syntactically; there are data structures that are well-behaved but the reasons for this are subtle (like "we got this RefCell counter right"). unsafe {} is a mechanism saying "even though this looks fishy, I promise that this function actually is well-behaved"; whereas unsafe fn is an indication saying "this function is well-behaved only if some non-machine-checked preconditions are met".
Never are you allowed to write an ill-behaved function. What you call trusted fn above, if I understand it correctly, applies to all functions. Violating documented properties when calling foreign code is morally equivalent to violating memory safety.