There's a big difference when discrimination is based on factors which a person can't control (ie, skin color, nation of origin, gender/sex, height, etc) and things intentionally displaying artifacts of offensive or objectionable political statements (ie, the swastika tattoo previously mentioned - represents a pretty powerful statement of political/cultural/racial superiority - nobody is born a Nazi).
I'm perfectly happy discriminating against fascists and potential criminals (ie, people who would just as soon cause me harm as otherwise). Makes my life a lot simpler and safer. That's a form of discrimination I'm okay with - not "you're just wearing the wrong color" - but if you're wearing red or blue bandanas in support of a criminal group - that's wrong.