Handling "misvoted" content

#1

Gilles raised an interesting point at the very beginning of the “What are we trying to build” thread about the problem of well-written but wrong content being upvoted above badly written correct content and how this is a limitation of the voting system. This is indeed a problem that SE doesn’t deal with well.

I propose that, assuming we do keep the basic ideas of i) up/downvoting, ii) a review queue system and iii) topic badges, we then institute a separate queue for this sort of thing. Something like this:

  1. I see a correct answer that’s lost in the noise OR a wrong answer that has risen to the top despite being wrong.
  2. If I have a “gold tag badge” or some codidact equivalent (we could set a tag-score threshold, for instance), that lets me cast a special vote or flag that will put this post into a dedicated review queue for this sort of thing.
  3. These queues will be organized by tags, so only tag experts can review posts in them.
  4. In this queue, these experts will be encouraged to edit the badly written but correct post so it improves.
  5. Alternatively, they can cast delete votes for the upvoted bad content.
  6. Perhaps the queue could result in specific markers being shown next to the post: “This answer has been marked as wrong/correct by our expert reviewers”

I’m thinking we should have some way for the most expert users in any particular sub-topic to be able to mark an answer as correct or wrong in a more official capacity. I have described one way, but I’m sure there are others. What seems clear to me, is that we will need to solve the issue of bad content with many upvotes and good content with too few, but it should be done in such a way as to not affect the real voting of the community. So this idea of “expert panels” sidesteps that.

8 Likes
Unlocking the upvote privilege
What are we trying to build?
MVP Proposal: User Trust and Reward System
#2

I like that; but as we start, there won’t be any gold-tag users, since those will have to be earned. The necessity this places on the presence of experts concerns me. Perhaps there’s a way to allow standard users to do the same thing, though perhaps with several more votes required?

#3

That’s a fair point, but on the other hand, this won’t be an issue for a while. It will only become a problem after we have enough content for this sort of issue to manifest. So by then, we will presumably also have our experts.

I’m thinking of this as an extra layer above what regular users can do. Regular users should still be able to edit and flag for mod review and mods could be able to shunt the post into the relevant “Expert queue”. We could also make it so that the expert queues only come into existence once a site and/or a tag has passed a certain threshold of participation.

3 Likes
#4

Because I don’t think all votes should be considered equal (like a bunch of 0 rep users who upvote everything vs a 5 year old account with 50000 rep and some gold badges in tags attached to the question), I think users should be able to very prominently see indications of votes by users more likely to be knowledgeable in this area of expertise.

Something like a special vote (in addition to normal up/downvote). Like a reaction icon with the user’s mini-badge showing their “expertise” level like if they have a gold badge or other earned distinctions.

Also I think attaching these special votes should be attached to a specific content version and visible in the revision history. So votes like “this is a correct answer, but is written poorly” is not permanently attached to the post after it was improved through editing. Such a mark can then be changed to just “this is a correct answer”.

1 Like
#5

I wonder how many of these problems we could avert if we allow votes to be withdrawn instead of locking them in after a few minutes like on SE. I understand the reason SE does it, to prevent voting abuse, but there are other ways to mitigate that I think, particularly as we look at answer scoring. If it’s even an issue. I get why it would be on SO, but I haven’t seen this kind of gaming on the smaller sites I frequent.

5 Likes
#6

I think that if we have an answer score, one user should contribute at most ±1 to that score. And if we have an “answer karma” (official presentation of the sum of votes on a user’s answers), it should just be the sum of the scores. But the default sort order should not be based purely on post score.

Stack Exchange already does this in a very limited way by allowing the asker to pull the answer of their choice to the top (accepted answer). This mechanism has unsurprisingly not aged well, since the accepted answer cannot be changed if the asker isn’t around.

The sort order should take three things into account:

  • The answer score.
  • The age of the answer, and perhaps of the votes. If a new answer gets 5 upvotes on its first day, it’s surely more useful than an answer that’s stayed at 5 upvotes for three years.
  • “Reactions” which are special upvote-like markers left by users.

On GitHub, reactions are a small set (currently 8) of “votes” that users can leave on a post. Facebook recently introduced a similar mechanism. Both show reactions as emoticons and show who left each reaction; I don’t necessarily propose to copy this behavior.

The basics of reactions in my mind:

  • Reactions do not affect score or karma/rep/EXP.
  • At least some reactions affect the default sort order.
  • The set of possible reactions is fixed on a given site.
  • Different reactions may have different prerequisites, for example ”non-scoring upvote“ would be available to anyone, “confirmed answer” would only be available to the asker, “an expert vouches for this” would require a certain score in one of the question’s tags.

(edit: This is a variation on @ user1306322’s earlier proposal which had escaped my conscious mind when I wrote this answer. Sorry user1306322.)

Some aspects may be customized per site:

  • Which reactions are enabled.
  • The impact of each reaction.
  • The prerequisites of each reaction.
  • The guidelines for giving each reaction.
  • The visual aspect of each reaction.

Some open questions:

  • Are reactions anonymous?
  • Are there negative reactions as well as positive?
  • Can reactions have influence beyond the sorting order and beyond appearing in the margin of a post, such as pushing a post into a queue? (I guess that’s implementable through a feed mechanism anyway.)

“An expert vouches for this” would be sort of like giving a bounty for an existing answer on SE, except that it works the other way round: influencing the sorting instead of giving more points.

4 Likes
MVP Proposal: User Trust and Reward System
Dealing With Obsolete Content
On the value of scores and votes
#7

I’ve always wondered if it would be an issue. AFAIK vote locking has been there since the beginning, so there’s no evidence that removing it would lead to abuse.

4 Likes
#8

Some answers on SO are highly voted but not just wrong, but dangerous. We need an “an expert says don’t do this” reaction, to be used judiciously.

4 Likes
#9

On DIY, generally speaking when something is dangerous, the professionals/experts (or serious amateurs like me) really do make it clear that a particular Answer (or what OP thinks they should do in the Question) is not a good idea. And there we are not talking about just “your code will have a security hole” but rather “someone could die”.

In other words, I think if there is a sufficiently active core community of experts, this should not be a problem. I suspect (but have no evidence to back this up) that the problem may be more prevalent on SO simply because it is way too big to manage properly - i.e., each individual expert can’t easily review “everything” on a regular basis they they can on DIY and many other smaller sites.

1 Like
#10

Expert reactions, either positive or negative, should not be anonymous. People should either stand behind those recommendations or not make them, and people evaluating those answers should know whom to ask for elaboration.

7 Likes
#11

Note that the fact that everyone can edit every post also means that an expert can edit in a warning right into the post. When a post is outright dangerous, that’s probably the best way to do it, because people finding the answer through internet search will likely not be able to interpret any standardized reaction (let alone the score), and may not read the comments either.

2 Likes
#12

That’s certainly allowed by the software, but seems to be taboo on SO. I’ve seen experts leave comments, but never edit.

1 Like
#13

It’s equally taboo in other stacks, from my experience, where the answerer often rolls the edit. As someone mentioned elsewhere, most times when someone wrote something, one will tend to resent any edits that alter the original meaning.

2 Likes
#14

Can we please stay on topic? Whether or not users on SE or SO welcome edits to their answers isn’t really relevant.

3 Likes
#15

One problem with this is that experts generally don’t like doing menial work.

As an expert on EE.SE, I rarely went thru review queues. I want to spend my limited volunteer time teaching and writing great answer, not being a janitor.

2 Likes
#16

Votes on SE are unlocked when edits are made. I know because I’ve occasionally downvoted, left a comment why, and said “ping me when it’s fixed, and I’ll undo the downvote”, and then subsequently sometimes removed the downvote after an edit.

(As a aside, it’s surprising how infrequently someone does actually fix a obvious problem and then pings me as directed. Sometimes they even fix the problem and not ping me, and I happen to discover it a year later.)

2 Likes
#17

No! Simple upvotes should be anonymous, but any other judgement someone else passes on a post needs to be explained and the source clearly identified.

Your reaction mechanism seems to be targeted for use by experts. If you don’t have enough conviction in your assessment, and aren’t willing to stand behind it and be prepared to justify it, then you aren’t really an expert (on that particular topic in that particular case).

As an expert, I would have no problem with this. The system gives my actions more weight, and in return I have more accountability. Sounds fair to me.

Some negative actions could be useful. These need to be done carefully and absolutely with accountability. One example that comes to mind is indicating a wrong answer that somehow ended up being popular and thereby erroneously upvoted by the unwashed masses. That’s rare, but I have seen it happen a few times. That’s a case where an expert stepping in saying, “No, this is wrong. Don’t listen to this.” would be useful.

2 Likes
#18

That helps when you want to retract a vote because the post has changed (like when the problem because of which you downvoted has been fixed). It does not help when you want to retract a vote because your knowledge has changed. For example, you upvoted a post, then later you discover a major flaw (or learn about it through the comments), then you would like to retract that upvote. Similarly, you may downvote because of a perceived flaw, but later you find out that it was a misunderstanding on your part, and want to retract that downvote.

3 Likes
#19

Sorry, but I disagree with this and would never do it. No matter how wrong, I (and everyone else) shouldn’t have the right to alter a user’s intended statement.

The correct solution is a rating system and comments.

3 Likes
#20

How is a statement like “this is dangerous” fundamentally different from a statement from “this has been answered at that place over there” which already is displayed right in the question when closed as duplicate?

It’s not as if the original text of the author would be defaced or something. It’s just a warning added to the top (and clearly marked as not part of the original text).

1 Like