Talking about what people should wear to Mass can be a dicey undertaking. Too often, it winds up as talking about what other people shouldn't wear, and that soil doesn't easily yield good fruit.
It can also wind up as telling other people what they should wear, which is often more rock than soil. "Fellow parishioner" is not a role conferring great authority in personal matters.
My guess is the reason people really don't like to be told how to dress for Mass is related to the reason that how people dress for Mass really is important. In a congregation, my appearance is what individuates me from everyone else; it's what makes me, in the eyes of someone else, this person rather than some other person, or no particular person at all. To a certain extent, saying I am wearing the wrong clothes is saying I am not choosing to be the person I ought to choose to be.
That may well be true, of course, but "You are not choosing to be the person you ought to choose to be" is not something most people have the standing to tell most people.
Another thing about conversations on how people dress at Mass is that, when people do give generalized advice, it often comes out sounding something like this: "Men, don't wear shorts. Women, don't dress like whores." That is, the question is treated as a question of propriety (or even of merely following a checklist) for men, but for women it becomes a question of modesty.
I fully realize modesty is not a besetting virtue of our culture, and that for a variety of reasons it's more of an issue for women and girls than for men and boys. But immodesty is just one reason clothes can be inappropriate for Mass, and I suspect using it to distinguish between women (and girls) and men (and boys) just gives people another reason not to listen. When the criticism directed at men is objective -- "You're wearing shorts!" -- and the criticism directed at women is subjective -- "You look like a tramp!" -- people may wonder whether there's more going on than an unbiased application of sound principles of propriety.