Then what mechanism will there be to allow those that consistently provide good answers to be seen as experts or the “smart person in the room”? This is ultimately very important to those experts. That’s why they are there. What are you going to give them that fulfills their needs?
We had two ideas:
- a trust level system (technically a more discrete reputation system) which will recognize users who are active on our site and produce high-quality content. This will lead to “global” privileges.
- a topic expert system (similar to SE gold tag badges) to recognize the experts in a given topic. This light lead to more specialized privileges in that topic area only.
That’s possibly not true though.
I’d consider this – Some thoughts on the direction of Buddhism.SE – from one of the founders of that site/community where he announced he was leaving. That author:
- Already has a real-world reputation in his field
- Presumably has little interest in on-site “reputation” (i.e. gamification and “ego”)
- Has other venues, online and in person, where he can teach one-on-one and/or broadcast
I think what he wanted was:
- A place to ask and answer expert-level questions from expert and semi-expert peers
- A place for well-researched answers – perhaps to narrow/specific questions not broad ones
- Not a replacement for Wikipedia or Google
And other experts such as university professors have, I believe, already their own and more meaningful ways of scoring professional “reputation”.
Based on some personal experience and reading between the lines of this other post – Buddhist forums & communities – one thing which appears to be crucial is the quality and consistency of the moderation.
So the content, the community’s guidelines, even social interactions.
That isn’t the say that SO’s use of voting, reputation, and privileges isn’t clever – I’m not sure it’s necessary though. If it were necessary then it’s even a disadvantage – it promotes longevity and volume of posts over expertise, someone who is genuinely expert but new to the site might find that unpleasant.
I think that reputation is a measure of how much you’ve used the site – and fear it might be a barrier to long-term or renewed growth (e.g. newer users, even newer experts).
3 Likes
Not that SE reputation does not really do a good job on telling whether you are an expert at anything but writing posts that gather a lot of upvotes. Note that posts that require a lot of expertise often gather relatively few upvotes because relatively few people are qualified to judge the post, and often also relatively few people are interested in the specific question. Also it has been noted earlier that you can get upvotes even on objectively wrong answers. Not to mention that you lose reputation if you award others a bounty, even if giving that bounty is exactly because you, as expert, recognized an under-appreciated answer as outstanding.
Also note that the site topic is typically much broader than anyone’s expertise. For example, consider an expert on a niche language on SO. It being a niche language, questions/answers on that won’t gather a lot of upvotes. Now let’s assume I start learning that niche language, and come across a question on SO which I think I can answer. The niche language expert also answers the same question. Now it is quite possible that I’ve got more reputation points on SE than that niche language expert, simply because I wrote posts about the vastly more popular language C++. It would be silly to conclude that my answer on that niche language question should be trusted more just because I have more reputation points. Rather the answer of the lower-reputation niche-language expert is far more trustworthy in that case; if our answers contradict, the probability is far higher that I’ve overlooked something than that the low-rep expert has.
7 Likes
Privileges are useful, but my point is really about public recognition.
I agree that the SE rep system was only a rough measure, but that’s fine. It was a single score shown wherever you went, that when high enough, made you stand out.
Put another way, if someone puts in enough work to write a lot of good posts, then they should be rewarded by some kind of “awesomeness rating”. Otherwise, what’s in it for them? And no, please don’t give me tired nonsense like “helping others”.
True, but good enough. You can’t get large amounts of SE rep without writing a lot of good posts. Yes sometimes simple things get upvoted out of proportion to their worth, and other really great posts only get 20-30 upvotes. The system isn’t perfect, but in the long run it’s good enough.
I’m not stuck on the exact mechanics of the SE rep system, but there still needs to be a way to get public recognition for writing a lot of good posts. Otherwise, there isn’t much point spending a lot of time on the site.
If you want experts to write a lot of good answers, you have to give them something in return. That something is public recognition above other ordinary users in some form or another. The SE rep system worked in that sense.
Then arguably the site should be split.
But again, you’re just giving me excuses for not doing something just because it’s not perfect. The SE rep system wasn’t perfect either, but at least it existed.
You are misunderstanding the point of the system. Yes, to some extent it helps users get a rough idea how trustworthy your answer might be. As you say it’s not perfect. However, the real purpose is to reward those that write a lot of good answers with public recognition.
Don’t let perfect get in the way of good enough.
Strongly agree. We do not need to nail down every last detail before we start. We need to have a reasonable starting point (which I believe we have), good communication as we work, and flexibility. We need to show progress sooner rather than later if we hope to be an option for communities that are in trouble now. So long as we work with those communities, we can all benefit – we get valuable feedback and analytics early, and they get something they can use and influence.
4 Likes
A single reputation number doesn’t tell you much, as already noted. Maybe expertise is better conveyed by listing the number of positive-scoring answers a person has written. Maybe it’s better conveyed by some measure of tag depth. Maybe it’s something else. We don’t have to decide this right now; we can work with our first communities and see what arises in practice. Since we’re not tying privileges to it but rather to site activity, it might not be as important as some think. Or it might be critical for gamification and engagement.
On some SE sites, a high rep might mean expertise or might mean “hit the HNQ jackpot”. High rep from one great answer means something different from high rep from a hundred answers with an average score of 2.
2 Likes
Exactly. You need something to keep the givers engaged. This is not optional.
Hitting the Hot Network Questions jackpot a few time does not give you high rep. HNQ is short lived, so the rep cap guarantees that. Even 5 such questions getting the rep cap for two days each is only 2000 rep.
There are anomalies, but they don’t add up to much. In the end, high rep can only come from long term consistent writing of good answers. That’s exactly what you want to encourage, and therefore need to reward.
You don’t get high rep from any one great answer. I just checked, and the most I ever got from any one answer was 390. I got 2035 from my top 10 answers on EE.SE. This rep wasn’t earned all at once. It came trickling in over years. You don’t get high rep from one or even a few lucky answers.
100 answers with 2 upvotes each is also only 2000 rep. You don’t get high rep from a moderate number of moderate answers either.
Yes, you do. You are underestimating how important this is for experts. If there isn’t some workable reward system in place at the start, they will tune out and not come back later to find you added one.
1 Like
You keep asserting that, but just because that’s what you need doesn’t mean it’s globally true. I’m saying we don’t know. According to another post here, on Hinduism having rep was a deterrent.
As for HNQ, I have seen people get several thousand rep from a single hot answer on Workplace. I personal rep-capped every day for close to two months on one site (granted, Meta) from about four posts. Just because you haven’t seen it doesn’t mean it doesn’t happen.
We are not going to do rep just on your say-so. We are not going to refuse to ever do rep just based on mine. I don’t actually have a strong opinion right now, but I do feel we do not need to do this in MVP precisely because we are conflicted about it. Once we give it, it will be hard to take away. There is no pressing need to give it. If that means that certain communities of certain types of experts find us unappealing, maybe that means they shouldn’t join at the MVP stage and instead wait for us to figure it out.
4 Likes
@cellio’s point in saying this was (I believe - correct me if I’m wrong, of course) that we don’t need to decide this right now because the point where we actually need to develop this feature is still some while away. We don’t need to have all features decided before we start developing, because we won’t be developing some of them for quite some time yet.
2 Likes
That’s a real shame. I was hoping to boot-start a new EE site here by providing high quality answers, and inviting several other experts I know to do the same. I’m not going to do that if the site isn’t going to give them what they want. They are going to tune me out if I keep coming back every few months saying “They fixed it. No, this time for real.”.
I realize that you all building this new site generally don’t identify with the people I’m talking about. In fact, I sense a certain hostility and they’re finally going to be put in their place attitude.
Unfortunately, that means you too easily dismiss the real value that makes a site successful, which is that it’s a good place to get your questions answered. That only happens when you have a resident core providing high quality answers. That’s the real asset of a Q&A site. Sure, you need people coming and asking questions, but that happens by itself once a site is known for good answers. In fact, the problem such successful sites have is that too many of the unwashed masses blurt out questions of low quality. The questioners will keep coming because you have something they want (good answers). But what about the answerers? What exactly will be in MVP that they want?
Also keep in mind that nobody ever does anything truly altruistic. They do it for their own reasons. One of those reasons might be the good feeling they get when helping others. Ultimately, all motivations are selfish. That’s fine, and does work in various ways in real life. Frankly, “good feeling from helping others” might be worth the occasional visit and answer, but it’s not enough to spend hours of free time writing high quality answers.
I’m not sure whether the problem is that you don’t believe me, that you think I’m misrepresenting the motivations of other experts, or that you feel that’s not how it should be. My impression is that there’s some of the latter going on. If so, you need to get over that if you want successful sites.
OK, that’s fine as long as it’s included in the first roll out.
Perhaps I misunderstood. I thought “we don’t have to decide that right now” implied “because it’s not going to be in MVP”. I apologize if I got that wrong.
This is something you absolutely will need to address at some point. The basic framework needs to be at least thought about, else you run the risk of implementing something where this is hard to add on. At least you identify the hooks now. Then you can leave detailed implementation to later. However, that “later” must be before first release.
Look, Olin wrote,
Now I’m canonically inclined to classify that as a form of pride or self-view – i.e. Māna – because, any form of comparison is (canonically) a type of “conceit”, which, is an experience for everyone (even including semi-enlightened people).
It (and encouraging or pandering to it) may be disadvantageous – e.g. from Wikipedia’s intro:
Māna (Sanskrit, Pali; Tibetan: nga rgyal ) is a Buddhist term that may be translated as “pride”, “arrogance”, or “conceit”. It is defined as an inflated mind that makes whatever is suitable, such as wealth or learning, to be the foundation of pride. It creates the basis for disrespecting others and for the occurrence of suffering.
It’s to the OP’s credit that he will post answers like this in which no “disrespecting others” is evident.
It’s also imo greatly to the credit of SE’s format – i.e. independent answers without “cross-talk” between users – impersonal.
Even so I imagine that we’ve possibly each seen sometimes, elsewhere, occasions where it seemed to foster in some people elitism, competition, non-cooperation/sharing, perhaps even rank unfriendliness.
But to contradict you, Monica, I think it probably is “globally true” – a universal characteristic of everyone but the most perfectly enlightened. I’m just not sure it’s an entirely good thing, though. And I am pretty sure it’s not the only thing, i.e. there are other motives for participating (in an intellectual topic and/or in a society) – even if Olin himself finds it a powerful and allegedly his only motive.
I gather you might keep something like “likes”, or even voting on questions? Maybe that’s still an incentive and feedback – i.e. “to provide the best answer in the topic/room” – even if you don’t then sum the votes and carry that sum around like a badge.
For a start, Discourse (e.g. here) doesn’t show your total reputation everywhere it displays your gravatar, yet people use this (here and elsewhere).
And I do think it might hinder some people (newcomers) just as it incentivises other (incumbents), regardless of whether they’re genuinely “the smartest person in the room”.
Keeping some statistics might be possible for those who are keeping score – just like here, perhaps:
I think it’s right to say it depends on the community. I feel that it’s an active hindrance or disincentive (even a barrier) on Buddhism.SE and I think that on the whole that community might have voted to do without it if we could.
Anyway I think it’s an aspect of “social engineering”, if that’s what that’s called.
Perhaps more to the point and not that I’m an expert, but I feel that “reputation” if you want it can be implemented as an orthogonal/optional add-one to the basic question-and-answer framework – and therefore I see no technical reason (though some might argue for there being a social reason) why it ought to be implemented in the first “MVP”.
On the other hand I’m not quite sure what “an MVP which we’re not embarrassed to release” means – i.e. how “feature rich” that’s meant to imply, therefore how many kitchen sinks to include in the initial cut.
That’s currently under discussion. That you think we must have some form of reputation in MVP is one opinion, but as others have said - the necessity of reputation as a form of recognition for experts is debatable, whether you personally agree with it or not. We’ve had another subject matter expert here on this very forum express a strong opinion that we shouldn’t have any form of rep, at all - so it’s clearly not a clear-cut case.
What has been agreed so far:
- We will have some sort of experience-based privilege system.
- We will have voting on answers; we’re undecided whether the scores will be displayed, but they will be used for sorting.
What that means is that if decide not to have reputation in MVP but later decide it should be there, we can add it in - we already have the votes and scores, so we can retroactively calculate reputation. If we decide to have reputation in MVP but later decide we don’t want it, it’s much more difficult to take it away than it would be if it was never there in the first place.
3 Likes
And can you show us that this is what they want? It’s what you want, but is it widespread? How do experts feel about not being labeled experts until they build a body of work, even though they’re experts “outside”?
Can you show us why showing something about the person’s relevant participation, rather than a single round number that reflects inputs having nothing to do with specific expertise, doesn’t meet the need?
I could well imagine a “reputation” score being computed only based on answers. Which isn’t what SE does. But we have to work it out.
Not from me. Pushing back on unsupported demands – not even saying no, but asking for support for the claims – is not “hostility”. If this is a pattern, please show us. If you’re upset that we’re not building olin.codidact.com, well, sorry about that – we need to serve many needs here and that means compromise, flexibility, and prioritization.
You are mistaken. Please stop jumping to the most negative (and unsupported) interpretations of what you see. We expect people to presume good intent here, just as we will on the sites we host.
2 Likes
I’ve been on board with keeping meaningful, public statistics from the beginning. A single number doesn’t tell us anything, but information about the person’s contributions – this many answers, this many people helped, etc – is important.
And I think we’re agreed that we’ll display the number of upvotes and downvotes on posts. We might not display a score, but we’ll give people the raw information so they can evaluate controversies themselves.
1 Like
On my first ever design project, after a year of learning the ropes alone, I was promoted to become the junior member of a two-person team.
They sent me to a week-long seminar to learn what the software design/modelling techniques were then – “structured design” it was called – and I came back enthusiastic.
Now my senior then, he was learning to be a team-leader – so he was interfacing with our .com, to get us test equipment scheduled for our project, and that kind of thing, budgets and what-have-you, maybe not the software design (which I was into).
Anyway, after defining the scope of the project (i.e. which elements of a protocol spec we were not going to implement support for), we had a meeting – just the two of us plus our “manager” in a meeting room – and I got all excited about explaining ‘my’ design on a whiteboard, like I could see it clearly in my mind and that it would work, that it was correct. And my team-leader was asking me questions., “but how would that design handle X?”, and I’d tell him, maybe impatient that he didn’t see it, that he was slow to agree, and eventually the meeting ended. And my vision of the design was accepted, that’s what we implemented and how.
But immediately after the meeting the manager said, an aside into my ear only, “It’s not enough to be right” I think that meant that I’d been a bit too enthusiastic about my own preconception, maybe not polite enough, and losing out on an opportunity for e.g. two-way conversation.
“Not enough to be right” – that was a long time ago now, but I found it pretty memorable.
Actually changing my character to have learned from the good advice takes longer, forgive me, still they say it happens in stages – first learn the theory (the “view”), and then practice.
To put an explicit counter point to Olin’s assertion that everyone participates just because of the reputation points, here a few things from my personal experience.
To start with, there was only one reason I ever cared about my reputation number on Stack Exchange: Because it unlocks privileges. That doesn’t mean I don’t care about what other people think about my posts; there are several posts where I check every now and then how many votes they already got. But that is the point: I care about those individual posts, not about the total reputation number. And I’m incredibly proud of one post on SO which was a late-comer, but over time raised to place 2 in the list, overtaking several high-voted answers (the top answer to that question is so incredibly high-voted that I doubt I’ll ever overtake it). But the total reputation number? Since it raised to give me all the privileges I care about, it’s no longer of any interest for me.
Also, about recognizing someone as expert: On math.SE I’ve learned to see Asaf Karagila as top expert in set theory. But I wouldn’t have been able to tell even in which ball park his reputation is. I now looked up because of this discussion; well, he’s actually at 4th place in the reputation list — but the point is, that is not why I learned to know him as set theory expert. Rather I learned to know him as set theory expert because I saw lots of good set-theory related answers by him.
There is exactly one case when I may look at the reputation number of someone, and that is when that person makes a statement about what should/should not be done on the site. If the reputation is high, I conclude that this person is experienced on the site, thus that claim carries weight. If the reputation is very low, I know it’s a newcomer without much experience on the site; so the claim is most likely just a personal opinion.
8 Likes