I don't have the canonical issue for this handy, but this is a known problem; type parameter defaults are only applied syntactically, when a type argument is missing in a place where it is required, not to resolve ambiguities in inference.
"fallback" is fundamentally hard in the same way that coercions are hard: when you're running type inference, you don't know where to add them.
(This is one of the costs of having a system where foo.into() works. In something like C++98 where each subexpression needs to be able to determine its own type in a standalone way, then you can look at the type it produced and see if that matches the type that's expected. But when things work bidirectionally it's just fundamentally harder. If done poorly you ruin compilation time because you have to try the full powerset of possible locations.)
Which, AFAIK, Swift actually does and has to give up in some cases because it can’t find a unique solution in a reasonable time but also can’t prove that one does not exist.