That would indeed be fair. I think what many people here may be forgetting is that this doesn't result in double taxation in practice, because you can deduct the taxes paid in your country of residence from the taxes due in your country of citizenship.

The only possible downside is the implied bureaucracy. Not all countries are good at minimizing that.