Here's my thinking.... you ride for YOU not for anyone else. Who cares what anyone else thinks? If you feel more confident wearing it, then wear it. If you want to protect your body, wear it. If you get hurt, YOU are the only one feeling the pain and paying the doctor bills. If you want the added protection, wear it. I love body armor BTW, very sexy.
On that note, I will wear different "outfits" depending on where I will ride and how "balanced" I feel that day. My husband was crashing quite a bit and started wearing some of my armor and he feels more confident and I feel better knowing he is better protected.