Should you care what others think of you? Should you care what EVERYone else thinks of you?
My view - I think there are people out there whose values you respect. Those people. Those people's views of you should matter. But if you don't respect the value system of a person, then why should you value their opinion of you? And, if you are in a situation where you think, "sure, but I need this person to have a high opinion so I can get a deal, a raise, a job" then I suggest you rethink your worklife path. Put yourself on a path where those types of compromises are not commonplace, because honestly - life's too short.
I just attended a funeral of a lovely 35 year old woman who has tragically left a hole in her community that loved her so much. She brought joy and service to many people in Chicago. I wonder her opinion of me (we knew each other, but casually). Her opinion would matter to me. Someone who measures success by the joy or help she brings others, not simply by the growth in personal networth. And quite frankly, sometimes one's success can be defined as much by who doesn't approve of you as much as who does.
I suppose what I am suggesting boils down to the a few of the old cliches. "To thy own self be true". Because ultimately the opinion that matters most is that of the person looking back at you in the mirror. Pause to reflect on your values. Focus on behavior that will make that person proud. Listen to the voices of others who share your value system - that feedback is key to keeping you on a path in which the man/woman in the mirror remains proud. And for other naysayers with different value systems... as my first boss would say - they can go hug a root.
(punchline thought here - external feedback matters - but it is not all equal. Seek out feedback from those with values you respect and admire. Don't fret about the rest).