It seems time we set some things straight. Being “PC” is not the same thing as infringing people’s right to free speech. Furthermore, it’s not nitpicky political correctness to criticize overtly racist acts like dressing in blackface, yelling the n-word at a group of black students or drawing a swastika in feces on a dorm.

The recent protests against racism at Yale and Mizzou have brought attention to these and other ongoing infringements of civil rights by those universities. Passions on both sides of the debate have escalated—one side wants to combat racism and the other wants to protect the rights of those who may or may not act in racist ways. Yes, Erika Christakis of Yale presented her thoughts about offensive Halloween costumes in a thoughtful, conscientious way. But she glosses over the issue central to the discussion when she says of her enjoyment of participating in other cultures: “Am I fetishizing and appropriating others’ cultural experiences? Probably.”

While I applaud Christakis for reflecting and encouraging others to discuss these kinds of issues, the trivialization and appropriation of others’ culture is a component to the white privilege that underlies all racial issues in America. She goes on to distract from the real issue of cultural appropriation with the typical fallacy about religious people being offended by provocative attire; religious people aren’t a marginalized group in society. It’s not about censoring anything that might offend someone—as one student told Christakis’s husband Nicholas, “It is not about creating an intellectual space! It is not! Do you understand that? It’s about creating a home here!”

What the University of Missouri system president Tim Wolfe told black students is far and away more callous and ignorant: “Systematic oppression is because you don’t believe that you have the equal opportunity for success.” And what’s really unfathomable is that so many people have criticized the students for protesting against this and other instances of racism instead of Wolfe.

No one likes to be told what to say. But when you’re saying something that marginalizes an entire group of people, it goes beyond the free expression of your ideas; it becomes a contributing factor to that marginalization.

The question thus arises of the role of free speech in institutions of higher education. Should all forms of expression be allowed on a college campus? The easy answer to this question is yes, and when it’s framed in this way many people would be likely to agree. But it gets trickier when one contemplates how violent or disruptive to the social and psychological security of some students this expression may be. Now, some would hold their view that speech rights should go unimpeded in this scenario, but others falter, noting that the security rights of the victimized students bear consideration as well.

The problem of rights that we face in the U.S. today is not the liberal crusade against free speech that racism-deniers proclaim it to be. It’s the fact that, in 2015, not only do we have a culture that perpetuates white privilege and racism, but those who make an effort to reverse this appalling reality are slandered as opponents of speech rights. So when people trying to change the status quo are shot down as “PC Police” or misguided protestors, it’s not conducive to the protection of rights that are due to all people. All it serves to do is to promote the objectively false notion that we’re living in a post-racial society where white people are being unfairly oppressed. And in that respect, attacking those who protest racism is equivalent to the racist actions themselves: It perpetuates the status quo in which it’s acceptable for minorities, especially black Americans, to be denied status equal to that of whites.

No one likes to be told what to say, but there seems to be a double standard. The systematic suppression of disadvantaged voices has become so ingrained in our culture that those who try to counteract it are seen as overreacting; but when those privileged by society feel as though that privilege is threatened, they go to great lengths to suppress those who threaten it. Kind of ironic, in a way.