I made one reference to a vote on an answer. I did that because it was so clearly and explicitly stating that this mode of frivolous voting exists (the person said they just voted because they saw other votes). But yes downvoting is an entire different story. On stats.SE it is completely out of balance with half of all downvotes coming from only ten people.
The other reference is about a question. While the discussion is about downvotes there were also closevotes.
In this discussion there is a lot of noise - talk is being directed to the principle of voting and other philosophical things. I call this noise and misdirection because in the end it is not at all strange to hold our judges, those which have the power to close, to account.
As an analogy: in taste panels it is extremely normal business (not the hedonic taste panels that answer ‘do you like this?’ but the descriptive taste panels that answer ‘how does this taste?’). If you want a consistent description then you need to train judges such that they are voting accurate and consistent.
For the close voting it is a bit mixed. It is on the one hand a way to have the community decide what they believe is good or bad (analogy with a hedonic taste panel), but at the same time there is also a (vague) concensus about what the community considers offtopic and ontopic and closevoting should reflect this community opinion (analogy with the descriptive taste panel).
Behind the suggestion here (giving some penalty for badly casted close votes) lies the fact that close votes are not always accurate or consistent. If there is a small group of people that hand out votes in a bad way then this is damaging the community. The overlapping question is how to deal with that, how to get rid of (or reduce) this vile environment where somebody like, say PewDiePie, would not be able to survive the heat.
My problem with downvoting is that people do not often return to the question to see whether it has changed. My current downvotes, which aren’t that many (but neither below avarage), are almost all on questions that have not yet changed. I actively check my downvotes once in a while to see whether something has changed and whether I can retract them (about half my downvotes have been retracted).
One problem is that people do not get a push request to reconsider their vote when a question has changed. Another problem is that downvoting is in disbalance with upvoting and caries much more weight (and at the same time it is very vague what it means). Because of these two points I simply hardly downvote at all anymore and use a comment to state whenever I find something in need of change or deletion.
Overall, from a statistical point of view, the up-/down-votes are relatively meaningless because there is too large variation among individuals how they vote and too little number of votes to reduce this variation by taking the mean of a large sample.
That’s very much a problem with the interface: You are not notified if a question that you downvoted is subsequently changed. You also cannot put a post on watch for changes (you can favourite a question, but that then is for the full Q&A page, not for the single post; also, it’s public information that you did so).
I think it would be a good idea to have something like a personal review queue. An edit gets onto that queue automatically if the post has a downvote from you (if minor edits are implemented, such minor edits may be exempt from that mechanism, so you don’t get notified about each spelling correction).
In that personal review queue, you would then be asked whether that edit resolves your issues. If so, the triggering downvote gets revoked. Otherwise, it stands.