主观性与客观性总是保持一致吗?基于Stack Overflow问题的案例研究
Do Subjectivity and Objectivity Always Agree? A Case Study with Stack Overflow Questions
AI总结:
本文针对Stack Overflow中主观投票与客观质量评估不一致的问题,使用250万问题和十种文本指标进行对比,并构建机器学习模型分类推荐与劝阻问题,准确率达76%-87%,优于现有模型。
摘要:
在Stack Overflow(SO)中,帖子(即问题和回答)的质量由用户通过投票机制进行主观评估。帖子所获得的净票数(赞成票 - 反对票)通常被视为对其质量的近似估计。然而,大约一半获得了有效解决方案的问题却得到了比赞成票更多的反对票。此外,大约18%的被采纳回答(即验证过的解决方案)也没有获得最高票数。所有这些违背直觉的发现使人们对SO所采用的评估机制的可靠性产生了怀疑。此外,许多用户对该评估提出了担忧,特别是针对其帖子的反对票。因此,为了确保无偏见且可靠的质量评估机制,对主观评估进行严格验证是非常必要的。在本文中,我们使用250万个问题和十种文本分析指标,将问题的主观评估与其客观评估进行了比较。根据我们的调查,四种客观指标与主观评估一致,两种不一致,一种既可能一致也可能不一致,其余三种既不一致也不分歧。随后,我们开发了机器学习模型来对被推荐和被劝阻的问题进行分类。我们的模型优于现有最先进的模型,最高准确率达到了约76% - 87%。
英文摘要:
In Stack Overflow (SO), the quality of posts (i.e., questions and answers) is subjectively evaluated by users through a voting mechanism. The net votes (upvotes - downvotes) obtained by a post are often considered an approximation of its quality. However, about half of the questions that received working solutions got more downvotes than upvotes. Furthermore, about 18% of the accepted answers (i.e., verified solutions) also do not score the maximum votes. All these counter-intuitive findings cast doubts on the reliability of the evaluation mechanism employed at SO. Moreover, many users raise concerns against the evaluation, especially downvotes to their posts. Therefore, rigorous verification of the subjective evaluation is highly warranted to ensure a non-biased and reliable quality assessment mechanism. In this paper, we compare the subjective assessment of questions with their objective assessment using 2.5 million questions and ten text analysis metrics. According to our investigation, four objective metrics agree with the subjective evaluation, two do not agree, one either agrees or disagrees, and the remaining three neither agree nor disagree with the subjective evaluation. We then develop machine learning models to classify the promoted and discouraged questions. Our models outperform the state-of-the-art models with a maximum of about 76% - 87% accuracy.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。