In struct literals this doesn’t seem ambiguous to me at all, in light of what C# does (as @scottmcm mentioned, new Foo { foo.bar.baz } is new Foo { baz = foo.bar.baz }). It’s the “innermost” thing that needs to match, and the whole expression is the thing that gets evaluated and stored. This behaves exactly the same as any other expression, while your alternative interpretation feels more like something you would expect from a pattern.
It’s the use in patterns that feels weird to me. I might get used to let Foo { &baz } = ... because it simply transplants existing pattern syntax into field position, preserving a useful property of patterns: it matches actual concrete structure in the object (a pointer). But let Foo { *bar } = ... is problematic because *bar is not a pattern: there is no “anti-pointer” in the object’s structure to match, and *bar can never be a full pattern anyway (that’s why it’s a binding mode spelled ref bar instead).