>you end up letting the most inconsiderate people with some of the worst decision making set the tone for society.

How's that different than letting complainers send state violence enforce stuff that's not a problem worth thinking about to most people?