Rating and moderating questions and askers

Codidact Platform Ideas
1

There are at least three major axes to evaluate a question.

  • Is is fit to be answered? I think we have a consensus that this should be handled by closure, details TBD. See the discussion around closure in the MVP.
  • How useful is it? How many visitors has the question helped?
  • How skillful is it? Did the asker do it well?

Discussions on Stack Exchange have an unfortunate tendency to conflate usefulness with skillfulness. But the two are very different.

If a student copy-pastes their homework assignment, that’s extremely low skillfulness. There’s no effort from the student’s part and there’s no reason to reward the student. But that doesn’t mean that the question is useless. The professor may have assigned a very insightful exercise about a widely applicable topic.

Conversely, a question asking why some code isn’t working, with a well-formatted reproducible example, all relevant context and error messages and so on, may be very skillful. But it’s rarely very useful, because no one else has the exact same code.

The questions on Stack Overflow that have helped me most are not the skillful ones. They’re typically short questions with little or no code, asking how to do some thing that isn’t straightforward from the documentation: low effort, high usefulness.

Stack Exchange mixes skillfulness and usefulness in question voting. I think this is harmful. Can we do better? What do we need to do now and what can wait?

5 Likes
2

I think there are two main types of askers.

Some askers want an answer NOW. They have homework to hand in, or they’re in over their head on their job. They don’t care about votes, or reputation. A majority of sockpuppet rings on Stack Overflow are created by people who want to work around a question ban or throttle: they don’t really care about the points, they just want to unblock their question, and maybe look better to answerers. Rating skillfulness is a way to rate those askers, figure out which ones are fine and which ones need help and which ones should just go away.

And then there are askers who want to be good participants. They put in some effort and they want to be appreciated. These users care both about their skillfulness and about their usefulness, and they are likely to be motivated by some number or badge.

In both cases, skillfulness is really about the person, so the skilfullness score of questions should be added, or averaged, or otherwise combined, and shown publicly somewhere in the user’s profile and perhaps in some kind of new question review queue.

Usefulness, on the other hand, is about each thread independently. And it’s somewhat shared with answers — the usefulness of a thread is really about its answers. Usefulness is rather like the “people helped” score on Stack Exchange.

Skillfulness is about rating the author, so it should not be influenced much by edits made by other people. Usefulness is about rating the thread and should not care who edited what.

3 Likes
3

Your reflections are interesting, I think you are up to something with separating the usefulness and skilfulness. This not only helps with things like sorting search results from usefulness, but also to determine how familiar a user is with the site from skilfulness for the purpose of privileges etc.

Maybe one could decouple them by keeping traditional voting buttons to measure the skilfulness (with an appropriate tooltip) and a “me too” button to get some proxy for the usefulness? Maybe also the number of page views could contribute to the usefulness.

3 Likes
4

I like the direction of thinking here, and separating out different components – usefulness/skilfulness – is a step in the right direction.

I would note that while it is not mentioned in the original post, these distinctions would obviously apply to answerers too. It is relatively trivial to post a link to documentation with a quoted excerpt; it is often more useful to explain why OP finds themself in a situation and how to diagnose/troubleshoot themselves. Providing a fish versus teaching someone to fish, to borrow a metaphor. Those sorts of answers aren’t always possible, but it would be excellent to be able to recognise didactic ability.

Not MVP, but it would be good to be able to rate edits. Most are trivial – spelling errors, inlining images, formatting etc – which don’t help the querent in themselves, but they help the site.

I like the idea of decoupling. I am sceptical about tooltips; to many, a big up arrow is ‘good’, a down arrow is ‘bad’. Perhaps clickable labels with (localised/translated) “Useful”, “Skilful/Well-Researched” to explicitly mark things as such. Whether we keep the traditional arrows in addition, is something to decide on- increasing numbers and a rep graph trending up does hit the ‘gamification’ buttons for some.

Interesting thought, but number of pageviews is most tangible useful to the site than anything! However, “usefulness to the site” is another metric we may want to explicitly track and reward.

As an aside, there’s also a parallel with good old Slashdot, whose commnents could be moderated up by one because they were “informative, insightful, or funny”, or down by one “troll, flamebait”. They were explicit about there being no “+1, agree”. The system was far from perfect but for a time it was a place where genuinely useful information could be found.

3 Likes
5

I notice that the Slashdot categories “Informative” and “Insightful” would also fit for SE answers: “Informative” would be for posts that basically tell existing knowledge (say, a post that cites from the relevant documentation, and shows how it applies in the given situation), “insightful” would be for clever solutions. Of course “funny” probably would not have a place on an Q&A.

1 Like
6

Aye! For the sake of completeness for those who are interested but unfamiliar, there were more (‘Interesting’, ‘Overrated’, ‘Underrated’, ‘Redundant’, ‘Offtopic’… think that’s them all); but I wanted to draw a parallel with the original suggestion.

7

I know; I was active on Slashdot for quite some time.

1 Like
8

I’m a little leery of having too many things for users to do (besides ask, answer and comment). I see enough on SE where:

  • People don’t Accept an Answer because they really aren’t sure how/when (as simple as it may seem)
  • People Accept an Answer but (based on timing of other votes or real clear if an Answer has 0 votes but Accepted) but don’t realize they should vote too.

etc.

Giving people 2 sections to “vote” might work. Giving them 3 or more - makes people think too much and I fear you would find “up on everything”, “down on everything” or “nothing at all”.

2 Likes
9

I agree, but it’s also a lot of overhead. Rather than rate all edits, I think there should be a way to reward particularly good edits. Basically some form of edit bounty. TBD whether it has to come from the author of the post, what it costs (if anything) and what the rewards are (a fixed award? A share of the answer’s “karma”? …). As you say, that’s not MVP, we can revisit that in a year or two.

I agree. There should only be a single action that you expect random visitors to do — a single upvote/downvote judgement. Anything else should come from special circumstances: accepting an answer (if we decide to go with this concept) only if you were the asker, special-case rewards like bounties only in special cases, judgements made through review queues…

2 Likes
10

I’d like to keep things as simple as possible for the majority of circumstances too. I had imagined the additional ‘ratings’ as an additional, optional interaction; eg either by a dropdown interaction on a post, or a set of user-selectable labels/boxes/tags, say underneath the post.

This was: [ Well-explained ] [ Useful to help me solve a problem]
[ Low effort ] [ Hard to understand ]

as examples off the top of my head.

Aye, something like that. As with the post thing I see it more as an optional tag/label/etc that someone can say "This was a useful edit because it [ added vital information from another source ] [ fixed the grammar, spelling and formatting of an impenetrable wall of text ] ", but with more concise labels.

2 Likes
11

I was thinking of the Slashdot model too. Being able to tag posts as not just “helpful” or “unhelpful” but as “insightful”, “well-researched”, “comprehensive”, or “incorrect”, “lacking-evidence” could help. I think such labels might be best as optional tags though, kind of like how in Facebook the :+1: is still the default with :heart:, :laughing:, :cry:, :angry: as extra options. Spam and “answers-different-question” (NaN) could even be included in the same system, as not all sites delete answers to other questions (which frustrates me to no end), while some sites could say that 3 “lacking-evidence” pushes it to a review queue just like a NaN would.

2 Likes
12

This is definitely not an MVP feature proposal, it’s pretty controversial for most sites, but I think it’s important enough to ask.

If you don’t know what “reactions” are on Discord, check the “Reactify” section on this help topic https://support.discordapp.com/hc/en-us/articles/207619737-Adding-Emoji-Magic

Problem: Post votes can mean anything, having a high or low score may be people saying something about you post’s quality, phrasing, message, whether someone (dis)agrees with it, or wants it to have more/less visibility by gaming the system.

People leave votes and don’t leave comments to explain them. Votes are anonymous (almost, you can still figure out if you look deeply into users’ activity, at least on SE).

Voting is quick, it takes just a second. Commenting takes time and may be bothersome, or people don’t want to have their username near their passerby quick opinion, or they don’t want to clutter the comment section.

Proposal: Reactions. About as quick as clicking the arrow up or down, but each reaction expresses a specific response like:

  • This is a high / low quality post;
  • I (dis)agree with this message - and this can be combined with the “high/low quality” reaction;
  • Funny - like on Steam, where reviews can be voted as “(un)helpful” and “funny”, it helps filter out funny but useless reviews;
  • (un)important issue - can be used to gauge public interest in pinning/featuring a post, like on Meta or for Hot Network Questions;

Reactions don’t take up valuable comment space, and wouldn’t be cleared when “nuking” comment threads, which helps preserve public sentiment/opinion on the posts.

I don’t know whether it would be a good idea to make it public which users added each reaction to avoid chilling effects. Maybe show it only to moderators.

This feature might not be great for most sites, it might be only useful for Metas, or it might only be desired by specific communities. In either case, I think when an open-source Q&A base site appears, I’ll be implementing this feature for me and my community based on the feedback that we’d all very much like it for our little game and anime club.

3 Likes
14

This is not a good idea and allow me to explain how. Votes collectively were enough to make people crazy/sad about their post quality. Reactions are just icing on the cake.
This is one of the things that I want to be different and want to have a really really unique idea on. The voting system is fine and it does it all for you, moderation and judging of content, what I was thinking is to dump this system and think fresh and here is what I thought of. Not rigorous
Only for newcomers

  1. There will less negative feedback on a post but more quick feedback that is made by the intent to improve upon a post.
  2. Create a role/duty that certain veteran and nice people hold, they will have the ability to add a bubble to posts so the OP can see and improve upon the post, the role/duty holders will help newcomers on the site. That’s a new and nice idea that I think can help welcome users across our site. The bubble will contain the necessary details that the users should do in order to achieve a good quality post, the recommended improvements/details will be provided by the user that has that particular role.
  3. Once the user updates their post, they can approve the bubble and it will go away.

This is a way, I think we can use to carry out and build a really good culture across our site WHILE carrying out effective moderation. This will also educate users to be reasonable on a Q&A site like codidact.

One thing that we can really improve upon is to how be welcoming with the ability to carry efficient moderation, which is hard I guess? Of course! but we can definitely improve from what SO was.

Implementing a solution that fixes most of the posts from the start is the way to go, instead of implementing a way that removes or takes care of bad quality post.

These suggestions don’t necessarily belong here but these are here to give you an idea of what I actually mean by opposing to @user1306322’s ideas.

2 Likes
15

Am I interpreting your suggestion correctly that you want to basically hide or do away with voting and instead switch to adding notice plaque / info bubbles / comments about how the post can be improved? If so, that is an interesting idea, and would be a big break from the whole reputation system.

Good posts would not need any requests for improvement, and bad posts would not get downvotes which affect poster morale. Also there would not be an incentive to post “popular”-designed posts to attract upvotes to satiate that internet karma hunger. I would like to explore this concept in more depth if someone else is interested in such an interpretation.

As for my proposition of adding extra “votes” in form of reactions on top of existing voting system I get that it could worsen the emotional response situation in some cases, but I would still like to have that experiment one day. Maybe not in this official version but in a fork.

4 Likes
16

Idea: keep upvote/downvote, and when you vote you get an additional, optional prompt to choose a reason from a canned list. I think most people are going to do one thing, but for cases where people are willing to provide more feedback (but not to write a comment), we could capture and display this additional signal.

4 Likes
17

In case we use anonymous votes (like SE), this would also have the advantage that the voter can give a reason without revealing the own identity (as would be the case with a comment).

However I’m not sure what you mean with an “optional prompt”. Do you mean one has to enable that prompt in the user preferences? Or that it would be a per-site option?

18

Optional prompt, as in, you don’t have to choose a reason…

19

OK, so not an optional prompt, but an unconditional prompt for optionally giving a reason.

20

Yes, I assumed that was implied by this:

21

Yeah, sorry, that’s what I meant. You get a prompt, and optionally you may provide more info.

Since writing that I’ve seen that SE is going toward reactions, which I remember being proposed here too. I don’t know how I feel about that, but it feels related.

1 Like