There seems to be a story there! I have my own experiences like that and can say with certainty that while it did end up with me having some trouble at work, I never once viewed speaking up for myself as a mistake. If you don't do it, who else will! Keep standing up for yourself, even if it leads to some conflict. You'll be better off, in the end, for it.