66

I recently learned that triage development has been abandoned midway:

Triage was predicated on us rewriting all of the views. Which... Very nearly happened. And then didn't.

(side note for readers who may find aborting development like that surprising: it isn't)

Given that originally planned restriction of views on troublesome question hasn't been fully implemented I would like to understand if whatever we've got has any effect at all (or all we've got was just a lousy blocking of VLQ flags because question is supposed to be under some toothless review).

For that, I would like to find statistics about closed and / or deleted questions that were posted in some representative period (for example in Spring 2018) and specifically, if a question being in triage had a noticeable impact on the average amount of views, answers, and comments. Learning if being triaged may somehow correlate with time-to-close (for closed questions) would also be interesting.

I expect these stats to help me roughly estimate if expected visibility restrictions were at least partially achieved.


Another related thing I would like to understand is if Triage somehow impacts repeat asking of poorly received questions ("recidivism"). My original expectation was that it might get substantially decreased but since the implementation wasn't complete I am not sure anymore.

In order to better learn about this I would like to get stats on how many askers of triaged questions (that end up closed or deleted) managed to ask their next question quickly enough (say, in 3 days or less after asking a triaged one) and how many of their next questions end up closed or deleted.

6
  • see also related request that attempts to learn about triage efficiency from a different angle - it is based on a different assumption (of a reasonably complete implementation of triage) and asks for stats that look much harder to obtain as it involves studying lots of details in question timelines
    – gnat
    Commented Oct 3, 2018 at 10:00
  • yes @Hans I am sure. As you said yourself "This is our web site, we made it successful by ourselves and they had nothing to do with it..." I want to keep maintaining SO as a knowledge repository and for that I want to better learn what tools we have and how much we can rely on these. They say triage handles ~20% new questions, that's quite a solid chunk and I want to understand how it works and if it works at all
    – gnat
    Commented Oct 3, 2018 at 10:23
  • 16
    Perhaps we need a "conspiracy" tag on meta :-)
    – Stephen C
    Commented Oct 3, 2018 at 15:09
  • 33
    The problem is, we can collate all the stats in the world. But SO have stopped responding to the core user base in any form other than ... words. Words are nice, but without action lead only to ... more words :S
    – jpp
    Commented Oct 3, 2018 at 17:54
  • 12
    @jpp I explained elsewhere that I depend on this site in my daily work - that's why I do what I can to keep it worthy for me. And that doesn't really depend on whether SO (the company) responds or not - no matter what they do (or don't do), I will be doing things I need anyway. If they provide stats I will try to find out how to use this data. If they ignore my request this will also be a bit of knowledge and I will try to figure how to leverage it. "I do what I can to keep SO content the way it is helpful to me..."
    – gnat
    Commented Oct 3, 2018 at 21:25
  • 13
    @gnat, I genuinely feel it's commendable what you are doing. It just would have a magnitude greater impact if SO Inc was on board with you and the rest of the moderation team (aka core user base).
    – jpp
    Commented Oct 3, 2018 at 22:13

1 Answer 1

14

I'm heading out on vacation for a couple weeks, but I wanted to try and address this before I left. To do that, I have to tell a story...

Return with me now to those thrilling days of yesteryear

Before we get to numbers, let's review the original motivation for Triage:

I'm gonna take a break from this timeline now, because... We could have stopped right there, in December of 2014. All the pieces were there: more powerful filtering on the homepage, triage to keep embarrassing stuff off the homepage, and a Help & Improvement queue to maybe allow some of the less embarrassing stuff to come back into play. Granted, all of them needed a lot more refinement - Triage alone took months to really "finish" - but the core ideas were there.

However... The homepage was an even bigger headache. At its core it was still driven by a mixture of tag preferences and magic machine learning; the filtering options were powerful but confusing. It wasn't very predictable or fast, and for some uses it scaled poorly. It looked and worked completely differently from all the other question lists. And finally, the Recommended homepage was one of four default homepages: the Interesting tab was still alive (though a good bit of the work on Recommended would be backported), the Active tab was used for all sites outside of Stack Overflow, and a variation on the Active tab (with fewer questions, built-in tag filters, and slightly different selection criteria) was used for anonymous readers outside of Stack Overflow.

Navigation was a mess. And... we weren't really making it better by tacking on yet another view. We'd already been discussing improvements for a while, but around the same time everything else described above was happening we'd begun thinking about this in a more structured way: how were folks using the site, and what could we do to make that work better?

  • Sam the Brand kicked things off with a discussion on meta after doing a bit of investigation via Google Analytics.

  • Then Sklivvz took and ran with it, diving deep into data on how users who answered questions navigated the site, tracking back via referers from when an answer was posted to identify the view the author had used to get there. By early 2015, he'd published a report internally, and the evidence was damning:

    1. The single biggest source of answers was Questions By Tag - Newest, the view you see when you manually filter by one or more tags.

    2. The next biggest source of answers, responsible for roughly half the answers of #1, was the homepage. Specifically, the Interesting tab.

    3. The third biggest source of answers, responsible for less than half of #2, was unknown: either the user's browser wasn't passing a referrer, or the referrer was external (cough Google cough).

    Sources trailed off pretty fast from there, but there was a bit of a long tail... Sklivvz, being the thorough sort, didn't stop with a list of referrers: he went on to calculate which sources were the most profitable for answerers as well (resulted in answers that got lots of votes): this demonstrated that sources like Unanswered By Tag, which were responsible for only a fraction of all answers, were none the less some of the best places to go looking for questions to answer if you wanted your answers to do well. The homepage was middling in this regard, probably only a little more profitable than answering questions you stumbled onto via Google.

    Finally, Sklivvz did a bunch of analysis on user behavior as it related to answering. Not surprisingly, he found that most users did not behave optimally when it came to earning reputation. We were doing a poor job of guiding answerers toward rewarding questions.

    The conclusion of this study led to...

  • the introduction of the New Navigation project in March of 2015. The goal here was to replace all question lists with a unified interface, easily customizable and consistent throughout. New Nav saw continual improvements throughout 2015, culminating in a Release Candidate.

  • Then New Nav was abandoned and eventually retired.

It's notable here that New Nav was not predicated on Triage, nor was Triage predicated on New Nav - they were separate projects with separate goals. But, thanks to Sklivvz' research, it is clear that Triage's goal - improving the homepage - was never going to be sufficient on its own; far more questions are answered outside of the homepage, where Triage had no effect. The only practical way to benefit the majority of answerers would have been to make this distinction irrelevant: and this is precisely what New Nav set out to do.

So what good is Triage?

Ok, now we can get to some numbers. From my perspective, it's still a better First Posts review than First Posts Review, and a better way of handling VLQ flags on questions than Low Quality review. But that doesn't directly help folks just trying to browse the site without bumping into lousy questions.

So let's focus on views. In May of 2018, 256,414 questions were posted on Stack Overflow. Of those, 31586 entered Triage, and 25508 were reviewed completely (that is a consensus was reached). The average view count across all questions posted in May was 89; the average view count for questions that entered Triage was 64; the average view count for questions that completed Triage was 67.

Broken down by Triage consensus:

Triage result Questions Avg Views
Looks Good 11379 76
Should Be Improved 9825 66
Unsalvageable 4299 48

Conclusion

Triage has not done everything we hoped it would, because our hopes were based on an understanding of how folks used the system that didn't reflect reality. In many ways, including but hardly limited to Triage, we've been unable to adjust the system such that it matches up with the reality of how folks are best able to use it effectively. Hopefully that will change.

However, Triage is not unsuccessful. Beyond its utility as a tool for moderation, it does at least partially fulfill the original goal: ensuring that fewer people have to look at terrible questions.

Related: Does Anyone Actually Visit Stack Overflow’s Home Page?

3
  • 2
    Interesting read Shog9, and thanks for the history lesson. Let me start off with wiahing you a pleasant vacation. On your post: I can't help to doubt your conclusion on the stats on triage. Because I wonder if there is any causality besides the correlation? As in active tags lousy questions pile up downvotes within seconds, and getting closed sometimes within minutes. So is it then user moderation that lowers the amount of views on a terrible question, or is it the triage review that limits the amount of views?
    – Luuklag
    Commented Oct 13, 2018 at 12:18
  • 2
    Triage is part of user moderation, @luuklag. It's unique among review queues in that eligible questions enter triage the instant they're posted; other queues have at least a sync delay. Triage walks new reviewers through closing, voting where appropriate. That's why I call it a better first post review than the First Posts queue. I could try to separate the effects of triage from similar questions, but that's a bit more work.
    – Shog9
    Commented Oct 14, 2018 at 16:12
  • 2
    I don't mean that there is no value in the triage review, so if that was what you took from my comment I'm sorry. What I meant was that I wouldn't draw the conclusion you did based on the data presented. To really "measure" the usefullness we would most likely turn triage off for a day or two and see if triage elegible questions gained more views during that time. Not an advice btw ;). What would be interesting to see is what percentage of posts in CV queue were sent there from triage. If I recall your flowchart correctly that is a pathway right?
    – Luuklag
    Commented Oct 14, 2018 at 21:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .