I agree. Being able to move/destructure the field is an important part of the feature; however, I think it's more important that the syntax reflects that the field can't be borrowed as &mut.
When someone sees the pub(&) syntax for the first time and wrongly assumes that the field can't be destructured, this probably won't cause any bugs, so it's not too much of a problem.
I should mention that I stole this idea from @kornel who proposed the idea and @RustyYato and @CAD97, who came up with the syntax.