StackOverflow reputation is broken
Last week StackOverflow announced changes in reputation system. This brought me back to my thoughts what makes this reputation system broken, and how it could be fixed.
I’ve been answering questions on StackOverflow for a couple of years now, on a daily basis.
For me, those questions are often opportunities to learn something new about technology. We tend to choose solutions that we’re considering “proven”. But that often limits our choice. So, when somebody asks a question along the lines of “How to integrate jOOQ and Vert.x” I say to myself: “Hey, that’s an interesting opportunity to learn something new, let’s dig in!”.
What makes this “question-driven” technology hunt even more interesting is that most of our technology now is open source. When somebody asks a question on StackOverflow about a certain project, you can actually get its code, open it in your favourite IDE and see how it works.
In general, the flow of StackOverflow works well. Except for one part — its reputation system.
There are few major flaws in it I’d like to discuss:
It just doesn’t add up
When you give an answer, first response to it decides its fate.
Upvote brings you +10 reputation, and downvote results in -2 reputation. But if your vote count is negative, then you don’t get any reputation.
So:
UPVOTE + DOWNVOTE = +8
DOWNVOTE + UPVOTE = 0Which is a weird math, and is inconsistent.
Solution: using constant values for upvotes and downvotes reputation changes. This could be +10/-10, +10/-5 or any other ratio, but is should be agnostic to the order of the votes.
Time spent asking and time spent answering
In StackOverflow terms, a vote is a vote. Vote for a good question is the same as a vote for a good answer.
Is asking questions hard? Most questions are asked as part of your work or studies. When something doesn’t work, instead of looking into documentation or sources, you just post this as a question and wait patiently for StackOverflow to do its magic.
Is answering questions hard? I claim it is. You need to get into person’s head. Figure out what the author is actually trying to solve. Understand what combination of technologies is being used. Translating from wrong terms to right ones. It’s a lot of fun, but it certainly takes a lot of time. Much more, than asking the question.
People asking the questions don’t actually expect an answer, “why this doesn’t work”, although it’s what they’re asking. They expect a solution. “Make it work for me”.
Still, upvote for a question gets you +10 reputation. Upvote for an answer — same +10.
Solution: splitting reputation into question-only and answer-only. There are people that ask important questions, and people that give good answers. They aren’t the same people. No reason to mash them together into same reputation system.
Marking as duplicate
Same questions are asked many times in a different manner on StackOverflow. Most common reason for that is difference in terminology. Someone will call “method” a “function”, or vice versa, for example.
There’s an option to mark question as duplicate, and specify what other question you should refer to. But even if the question was marked as a duplicate correctly, it doesn’t benefit the one that market it such in any way.
What this creates is an incentive to actually answer duplicate questions yourself, often by rephrasing the answer to the original question. Which creates noise in the system that should have produced signal.
Solution: providing reputation equal to correct answer for marking as a duplicate.
Outdated questions and answers
StackOverflow is full of outdated questions. Such as “how to filter a list in Java 6?”. Java 6 was released in 2006, and ended its life 6 years ago. Or questions about Oracle 8 (1997, yeah).
But questions and answers about Visual Basic, C# 3.0 and even Smalltalk are still scored the same, no matter how irrelevant they are. Which in turn creates situation of stagnation in the reputation system.
Solution: introduce reputation decay for inactive questions and answers. Now, inactive is a very vague term. But using even something as simple as views over time (which StackOverflow already tracks) would allow for outdated questions to sink down in terms of reputation, which in turn will float up questions that are more relevant.
Correct answer weight
That’s just some simple math. Providing a correct answer awards you with +15 points. Providing any answer may benefit you with +10 points. There’s a strange system now that the author of the question also usually upvotes the correct answer, rewarding +25 reputation points in total.
Solution: provide +25 points for marking the answer as correct. Remove that ability for the author of the question to upvote correct answer.
Commenting on a question
This is very similar to the duplicate questions problem. Often, the question being asked is trivial enough, that it could be solved by just writing a comment, instead of an answer. In some other cases, asking question in a comment brings the author to the correct solution. But apart from feeling you, this doesn’t bring any reputation benefits. Which in turn creates an incentive to provide an answer, which may be incorrect one.
Solution: there is an option to upvote comments. But it doesn’t benefit your reputation. Giving it some impact, +5 reputation, let’s say, should encourage people to provide solutions, no matter if it’s in the form of a comment or an answer.
Who decides what’s correct?
Currently, it’s the one that asked the question. This results is a number of flaws:
- “First to solve” is usually marked as correct. Remember that we’ve already discussed the difference between “an answer” and “a solution”.
- Additional answers are rarely re-evaluated
This creates some strange situations, when there’s an outdated answer marked as a correct one, with a more relevant answer having more upvotes beneath it.
This is a complicated situation, and I don’t see any simple solution to it.
In some situations most upvoted answer may not be the most “correct” one, just the most “understood” one.
In other situation, the question has more that one “correct” answer. Same problem may result from different causes, and may have different viable solutions. It’s just something all StackOverflow users should keep it mind, though.
