I agree, but that applies just as well to downvotes. I don't have good ideas how to run a required-comment-to-vote system, I just don't see the asymmetry between up and down.
Maybe I wasn't clear in saying that I don't think it should be necessary to comment in order to downvote, but I think it should be required to upvote a reply in order to downvote (or make your own reply).
