众包中投票数对主观语音质量评价信度和效度的影响
Impact of the Number of Votes on the Reliability and Validity of Subjective Speech Quality Assessment in the Crowdsourcing Approach
摘要：
传统上，根据ITU-T Rec.P.800，在受控的实验室环境中评估传输语音的主观质量。反过来，通过众包，众包工作者使用他们自己的收听设备，在他们自己的工作环境中参与主观的在线实验。尽管存在这些不太可控的条件，但越来越多地使用众包微任务平台进行质量评估任务，推动了对标准化方法的高需求，从而产生了ITU-T Rec.P.808。这项工作调查的可靠性和有效性的质量评级收集通过众包语音质量评估，作为ITU-T建议P.808的输入判断的数量的影响。在不同的平台上进行了三个众包实验，以评估三个不同的语音数据集的整体质量，使用绝对类别评级程序。对于每个数据集，平均意见得分（MOS）是使用不同数量的众包判断计算的。然后将结果与标准实验室实验中收集的MOS值进行比较，以评估众包方法作为投票数的函数的有效性。此外，平均分数的可靠性进行了分析，通过检查评分员之间的可靠性，在确定性的增益，和MOS的信心。结果提供了一个建议，每个条件所需的票数，并允许模拟其有效性和可靠性的影响。
英文摘要：
The subjective quality of transmitted speech is traditionally assessed in a controlled laboratory environment according to ITU-T Rec. P.800. In turn, with crowdsourcing, crowdworkers participate in a subjective online experiment using their own listening device, and in their own working environment. Despite such less controllable conditions, the increased use of crowdsourcing micro-task platforms for quality assessment tasks has pushed a high demand for standardized methods, resulting in ITU-T Rec. P.808. This work investigates the impact of the number of judgments on the reliability and the validity of quality ratings collected through crowdsourcing-based speech quality assessments, as an input to ITU-T Rec. P.808 . Three crowdsourcing experiments on different platforms were conducted to evaluate the overall quality of three different speech datasets, using the Absolute Category Rating procedure. For each dataset, the Mean Opinion Scores (MOS) are calculated using differing numbers of crowdsourcing judgements. Then the results are compared to MOS values collected in a standard laboratory experiment, to assess the validity of crowdsourcing approach as a function of number of votes. In addition, the reliability of the average scores is analyzed by checking inter-rater reliability, gain in certainty, and the confidence of the MOS. The results provide a suggestion on the required number of votes per condition, and allow to model its impact on validity and reliability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
