I strongly agree; the entire setup of marriage enshrined in law the way it currently is is just wrong. It discriminates against polygamists, as you note, any certainly other groups. The societal rights and burdens of marriage should definitely be available to arbitrary consenting parties.

Even more so then, that the genders of the parties involved are completely irrelevant.