MVP Discussion: Reputation

Codidact Platform Ideas
21

I think regular votings for (non-binding, simple) moderation privileges would be overkill and not possible.

In most democratic states the government (e.g. the cabinet) is elected through direct or parliamentary vote (and therefore directly legitimized by the people), however the administration is appointed in a work-like environment.

Even then regular simple-priv-elections would either need to be huge and therefore time-consuming to manage or would limit the amount of users allowed ot moderate. Even Stack Exchange, with a full-employed Community Management team is not able to run many elections at once (only one at a time). How should this work for a site, where for now everyone is a volunteer?

I think topic-expert-badges are a good idea and so is coupling them with moderation features for these posts. However badges are always also about awarding new users who are doing something good. Having only your proposal of badges would make them rare and that would remove some incentives for new users.

2 Likes
22

Point taken on the votes. The idea was to separate out kudos and badges.

Its possible for people to get lots of kudos but i dont necessarily think automatically giivng them mod privs just for that is a good idea. Thats why I mention some kind of gamified rewards system separate to badges.

In my concept, badges actually mean something. And they reward people focusing on one area well. Kudos could very well provide a user to do more actions but i personally think moderator actions should be divorced from the points system.

2 Likes
23

@Gilles makes an excellent point about different people being motivated by different things. There is an obvious inverse/converse to that too: different people are demotivated by different things.

For example, on SE: a simple, two sentence question can gain ‘Hot Network’ status, and attract hundreds of votes for it and its equally brief, off-the-cuff answers. Another question might go through various stages of refinements, with a well-researched, insightful answer and sources to back it up. Which then sits at zero, unaccepted. That sucks.

I am not saying that simple expenditure of time spent makes for a ‘better’ post, or length of post, or number of references it cites. But system that works more broadly than simple upvote/downvote points can arguably help motivate or pull the levers for those to contribute for more reasons than seeing a big number beside their name.

Perhaps this could be tied in with @mattjbrent’s stratification of user experience. Allow users to ‘react’ to / tag posts. Rather, or in addition to, than a simple up/down vote, users can tag it ‘Well-researched’, ‘Useful clarification’, ‘Insightful’. (All examples off the top of my head, or stolen from /. in the last instance)

This would come at a cost: complexity. Voting up/down is easy (ish, eternal debates on Meta about what constitutes acceptable voting practices aside). Nuanced curation and broader recognition of useful content takes more effort, but should reap greater rewards in respect of engagement, and people creating useful content because they are rewarded for doing so.

7 Likes
24

There’s a lot of very good discussion here. Most of it is for the medium term though, not for the MVP.

For the MVP, I maintain that it boils down to one thing: no reputation, in the sense that there must not be a single number that everybody focuses on. We can have multiple numbers, which dilutes the importance of each number and lets us vary the practical impact of each number (displayed discreetly? displayed prominently? Determines privileges? …). If someone wants to boost their total upvote count, ok. But don’t base everything on that, and if someone else prefers to boost their edit count, let them. But even with less prominent numbers, tread carefully: some badge hunters on SE cause a lot of harm.

Where Stack Exchange uses reputation for privileges, I think we should gradually evolve competence-based privileges. Something like Discourse for initial familiarity (analog to the first 50 or 100 rep on SE). Then, for things like editing, closing, deleting, maintaining tags and so on, use some measure that’s more tied to competence, ability or experience specific to that particular activity.

No, I don’t see such a consensus. I disagree and I’ve observed I’m far from being the only one. If this turns into SE but with different management, I’ll participate with content and moderation the way I do or did on SE, but I’m not particularly interested in building it. If this is a repository of knowledge, designed with the experience gained from a decade of SE, I’m much more interested.

The friction of moving to Codidact should be minimal for basic users: read-only visitors, question askers and answerers. It doesn’t need to be minimal for power users.

5 Likes
25

So, it seems like we’ve basically reached a consensus of no reputation then. That raises the question - how do we handle privileges and deciding when users are deserving of privileges?

My suggestion: we do a system based on “stars”. There are gold, silver, and bronze stars, which can be earned by completing helpful tasks around the site. Each privilege requires x stars from each category to earn the privilege.

Of course we’d have to work out the specifics of exactly how we would use the ranked star system for different privileges, but that’s in the future.

Also another point to addd is that in this system we could potentially make different colored stars for different types of participation, i.e one for editing, reviewing, asking, answering, meta-ing, etc. That could potentially cater to our needs a bit better than generic rep.

26

I apologize if I missed it in this discussion, but: do not conflate reputation and voting. The former is generally derived from the latter, but not doing reputation does not mean not doing voting. Voting is essential, to show visitors which answers have community support and which do not. Secondarily, votes also serve as a marker for “I’ve been here already”, for navigating questions with lots of answers.

We don’t have to compute and display user reputation for MVP, and if we’re not sure how we want the rep system to work then we certainly shouldn’t do it early and then change it (too much angst). But we do have to have voting on posts.

9 Likes
27

I must say that the voting on comments works very well for me to quickly see the important comments.

But for questions and answers…

For answers it is just a kudos-system. There is not much meaningfull differentiation to make between different answes (there aren’t, on average, many answers per questions) except in the few rare cases when questions have many multiple answers.

But then (when there are many answers) it can still fail a lot. For instance, when a new diamond is created as response to an old question with lots of sandy answers. It basically locks any question (reduces motivation to improve the q&a) after a certain time because new answers will gain much less votes.

(it’s not for nothing that some people tend to jump to questions with quick answers, but they might be short and unresearched and possibly including code that doesn’t even work perfectly because it hasn’t been checked, that is gaming the system, fastest gun, which becomes a problem to a collaborative community.

Voting (the psychological way it works) is creating an incentive to respond quickly and this influences the quality of answers, or at least it did influence me, even on a relatively slow q&a site like CrossValidated, to answer more fast than best. And from another angle it made me demotivated to answer old questions on physics/astronomy where I obtained less votes than on CV eventhough I am a physicist and not a statistician. )

For questions it may differentiate between more or less important questions (at least in terms of popularity). But I believe there is a large variance (source of error) in the rating of equally important questions. Especially most questions (a large majority) tend to fall in the categories of 0 score or 1 score, and there isn’t that much differentiation going on. The relative fraction of 0 and 1 score (determined after 30 days) questions is increasing. The absolute number of new questions with score >1 (after 30 days) is decreasing. It is not a sustainable system.

Anyway, whatever one may find about voting and reputation, possibly it might be something that should vary from community to community depending on their needs. Some care more some care less.

Then basic voting should be MVP but it might be nice to design it in a modular/optional way, such that each site/community can decide about it and include it or not (and add their own flavour of reputation to it if required). To me it doesn’t seem neccesary for a Q&A site, however it may be neccesary on order to build the community that is behind the site.

I believe that this might be an important consideration for the MVP. For who are you creating this software? I guess that this codidact might become a new standard for Q&A (if sufficient energy is put into it; then these Q&A websites might pop up like websites running CMS systems like Jooml/Drupal/etc or blogging software like WordPress and it can become a new genre) and then flexibility should be an important part of the design in order to be suitable for many different communities.

3 Likes
28

Both instagram and twitter are experimenting with getting rid of ‘likes’ or hiding them in some way.

This internet-based reputation system is too much alien to real-life communities in order to be a good reflection of the (tighter) communities that we have more commonly in real life.

(I do not want to say that 1: tight communities don’t exist on the internet; but when they do exist, then they, the bonds that held the community together, are often based on conversation and not on points. 2: neither is some way of commitment and ‘scoring’ not equally important in complex real-life communities, but it comes more naturally and it is more complex than simplistic voting).

Being ‘high in rank’ and part of a community only/mostly because of ‘rep points or likes’ will in the long run be detrimental for the overall motivation that runs the community. The quality of the Q&A/posts will resemble the quality of the voting system. When voting becomes a popularity contest then created answers and questions will become a popularity contest (this is good for a commercial website which just wants to become popular; and popularity helps to make money; but one might wonder whether this is the goal for a website that has a different type of government which is not oriented towards making business and making profit/money).

Voting systems have been created in the last two decades and made use of the ‘power of the mass’. When they started it made use of a (democratic sort of) power that had not been used before at the same scale. And because of that they became very successful (Large scale organizations don’t need to care about the level of quality; as long as it, mediocre quality, suits an enormous amount of people then one doesn’t need to care).

However, this ‘mass’ is getting more and more tired of the pitfalls and negative side effects associated to the simple voting and popularity based content. New systems need to be able to differentiate between different users and not make everything revolve around popularity and the ‘average’ popularity choice.

If you want a StackExchange copy to be viable (get close to or even beat StackExchange) then it should not try to fight the popularity contest with StackExchange. A new (successful) StackExchange should be differentiating in many ways (not just differrent people that run it) and the orientation of the community is one way to do it.

Now, to kick start the project, one could start with the Reputation and Voting, but it would be nice to remain open to the large variability of potential ‘customers’ that may wish to use codidact software. One of the type of groups that may wish to use codidact are a community that is more experienced/professional and cares less about the voting and reputation or may even want to stay away from it as far as possible.

2 Likes
29

Some interesting data from StackExchange. When you look at voting activity then indeed it seems like old posts do not get much attention (anymore). So the whole dynamics of the voting is (currently) solely based on contributors/users looking for new posts and rating them. It almost seems like the business of StackExchange is more into the rating of new stuff rather than making the old stuff usefull.

Screenshot_2019-11-15%20fraction%20of%20early%20votes%20per%20month%20-%20Stack%20Exchange%20Data%20Explorer
Screenshot_2019-11-15%20fraction%20of%20early%20votes%20per%20month%20-%20Stack%20Exchange%20Data%20Explorer696×399 66 KB

In the data that made this graph earlyVote means a vote on a post (Question/Answers) within 7 days of the creation date of the (parent) Question.

You can change this in many ways (look at the creation date of the answers as well or reduce 7 days down to 1 day) and it doesn’t change the image that much. On CrossValidated apparently 80% of all the votes are made within 1 day after the creation of the question (I suspect that this is mostly votes that occur when questions are in the top list).

So whatever happened during the the first day after a question got posted is extremely important for the score that it receives. It should be possible to optimize this and make it possible that questions receive more (voting) attention rather than only during this first day period.

Also interresting is this curve graph: the average score of an answer as function of the time between the answer and question.

On CrossValidated an answer that gets posted within one hour that the question got posted receives on average a score of 4. This drops down by 25% within 24 hours to a coarse baseline of a score around 3.

There are some interesting peaks (for instance after 13 days). This might possibly be because the software bumps up questions after some regular interval. But the volume of these peaks/answers is not so large. About half of all the answers are posted within 4 hours that the question got posted.

image

The image below is for StackOverflow which has more data and shows less noise in the points. An interesting 24h pattern becomes apparent.

mean answer score as function of response time

The view below, with the average score taken as function of the day distance (instead of the hour), allows to view the curve on a large range. It seems like very early answers score well (and these are the most numerous), but on average the very late answers score better.

But it takes 90 days before some maximum is reached. Between 0 and 90 days either questions tend to attract relatively more lower quality answers or good quality answers tend to attract less votes (or a mixture of both).

mean answer score as function of response time

4 Likes
30

Which gets to “why vote on questions at all”? Actually, I am in in favor of some level of voting on questions. It can be useful, and I am (in general) in favor of votes & rep etc. However, I posit that “most” (hard to say for sure) people are typically in one of a few modes:

  • Looking for help/asking - If I’m one of those, I don’t care how the other questions did with votes, I just want MY question answered. Quickly. Authoritatively.
  • Topic matter specialists - Not necessarily true experts, might be hobbyists or just “interested”. But these people will look at all questions within their domain. The votes won’t matter. They might skip a question altogether based on the title but “0 votes” won’t deter them. Even “-2” won’t deter them - they’ll look at the question to see if it can be cleaned up or (if not) VTC.
  • Browsers - Due to HNQ (i.e., not usually on this topic site) or any other reason (perhaps looking at all questions with a tag matching their question, or whatever). These people also likely won’t care about the Q votes.

In the end, I believe Answer votes matter - they show a visitor “what’s the best” (or at least most popular, I know they’re not always the same) so they can look at the results and get useful information more quickly. They also, subject to debate, reward the people who wrote the answers (whether psychologically or with actual benefits like more privileges). Question votes do very little for the site or for the users except make the askers “feel good”.

For both practical reasons (simpler code) and user reasons (avoid confusion), I think all topic sites within a Codidact instance should function the same. The value of each “thing”, the reputation needed for privileges (if we do that) or badges (if we do that) can vary, as it does to a limited degree on SE. The “owners” of each topical site (moderators and/or high rep users) should get to make those decisions, though arguably some limits should be placed on frequency of changes, etc.

1 Like
31

Is codidact gonna be a (general) open source software to run q&a websites (with lots of variation and options like an open source web content management system) or is it gonna be a software for a specific website (which is also open and may be freely used by others, but that is not the goal itselve)?

32

There are two, technically separate but very closely linked projects:

  • Codidact - An open-source (anyone can copy, fork, install on their own server as they please (within licensing TBD, but will definitely be towards the “open” side of things). The “technical” discussions are about this product.
  • A specific instance of Codidact, name TBD - A community-run (as opposed to “corporate”) instance of Codidact which will support communities similar to those currently on SE. The “policy” discussions are about this particular instance.
33

It starts to become a bit confusing to me (which stakeholders are involved). There are several topics on this forum that discuss details/properties of a final or minimal product. But, ‘we’ haven’t actually defined the user very well.

We are writing down all kinds of good ideas but it is not very user focused.

34

@MartijnWeterings I know. It is confusing. The group of people currently involved are spearheading both parts of this at the same time. The needs of the “instance of Codidact” are, appropriately, defining the initial requirements (MVP) for Codidact. At the same time, we want this to be open (in the “welcoming” sense, though defined a bit differently from SE) to new users, casual users, etc. who are (pretty much by definition) not the group of people doing the development. And trying to balance years of experience on all sides of SE and other Q&A platforms. And trying to do it all quickly!

35

I think awarding people who ask questions is really important. Good questions (and especially good titles) are what targets traffic to the site. Bad questions mean that the hit rate is lower, and that people are less likely to get what they wanted from search results. Voting is the easiest way of gamifying this, but not the only one.

Perhaps this is something that you could experiment with but I dont think it would be wise to make this a first cut thing as voting is tried and true.

6 Likes
36

I think it would help the discussion to keep several aspects of the topic separate. In the following I list what I consider the importatnt points, together with my opinions to it.

  1. Should we implement voting on questions/answers?

    I think there’s consensus here that this should be the case at least for answers; I think question votes are useful, too, but have no strong opinion on that one.

  2. Should there be some point system that connects the upvotes/downvotes you get (and other things you do on the site) with priviledges on the site?

    I think yes.

  3. Should there be a single number (like reputation on SE), or a set of different numbers based on different activities?

    I would vote for a set of different numbers, corresponding to different aspects of contributing to the site.

    Note that even on SE, there are actually several numbers; except that reputation is a catch-all for most things, and the other numbers (like number of helpful flags) are very specific.

  4. Should those points be publically displayed, publicly displayed only as levels, or not displayed at all?

    I think it is useful to see some indication whether another user is experienced with a site, not so much when considering answers, but when considering comments on what you should/shouldn’t do. If a high-reputation user tells me “don’t do this” I’m going to assume that this is (possibly unwritten) site policy unless I’ve got indication against it. If someone with low repuation makes the same comment, I’m going to assume that’s just a personal opinion.

    But for that, I don’t need to have an actual number. A level indication (like the milestones in the SE priviledge system) would be wholly sufficient.

    About my own points, I couldn’t care less whether others can see them.

    But then, that’s how I personally treat those points; of course the decision needs to come from what profits the site most, and I might be just an outlier (although I certainly hope that I’m the type of user we want to attract).

  5. Should those points be displayed to the user that got them as number, as level, or not at all?

    Of course the user should not be shown less than the general public. But the user might get shown more.

    I think the user should get shown the numbers in full. Note that numbers that are shown to you, but not anybody else, are not very useful for bragging, therefore I think people are not likely to optimize those for ”bragging rights”.

2 Likes
37

It seems like we’re close to reaching a consensus here so I want to point out the solution I offered (which seemed to be highly liked by the community). Is everybody okay with a system like this? I tried to include all the points that we covered in this thread.

1 Like
38

An idea about the question votes:

Maybe it would be a good idea if question posters would get some points (reputation) on upvotes on answers to their question. That is, they get rewarded for asking a question that gathers good answers. A question that gathers good answers probably was a good question.

Of course, the points you get to upvotes on your own answers should be larger than the points you get on answers to your question; maybe if the answer poster gets 10 points per upvote (as currently on SO), the question poster gets 2 points per answer upvote.

4 Likes
39

Question votes and reputation are different topics. Let’s discuss question votes in their own thread.

1 Like
40

I agree with @gilles that for MVP we shouldn’t display a single reputation number, because once we give it we’ll never, ever be able to take it away again and, demonstrably, we’re not clear yet on what we really want.

Not MVP: I find myself thinking in terms of a set of “stats” on the user profile page, something like:

Questions: 147 posted, 592 total votes, newest (date), top tags: (list)
Answers: 365 posted, 2161 total votes, newest (date), top tags: (list)
Edits: 79, newest (date) [these will be edits to other people’s posts, not yours)
…

This sort of thing gives people a sense of the user’s activity, and also feeds into the idea of privileges based on related activity rather than a total score. It could also feed into something like situational flair later – when a user with X total score in a tag answers a question in that tag, add some sort of doohickey below the user name to convey “subject-matter expert” in that context. Not saying we should necessarily do it, but this approach could enable it if we want.

With the profile also linking to all of a user’s activity, anybody who wants to take a deeper dive can do so – but we recognize the multi-faceted nature of a user from the start and don’t try to summarize it in one number.

If, later, we decide we want one number, it’s easy enough to add at that time.

6 Likes