If each one came to that decision separately, then sure, ban them. The problem arises when a company controls, say, 90% of the restaurants. And then ban you for no reason.
It's not a problem if people frequent the popular restaurants by choice. Maybe regular people aren't fans of restaurants whose main differentiating feature is their "swastika shirts welcome" sign.
So when Twitter starts to ban people for no reason, let's object then. The idea that we all have to start when they are banning Nazi's because of some slippery slope is ludicrous.
