arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1808.06152stat.MEcs.LGstat.ML
收藏

体验调查中问题表征题的设计

On Design of Problem Token Questions in Quality of Experience Surveys

Microsoft Corporation(微软公司);Facebook(脸书)

Jayant Gupchup, Ebrahim Beyrami, Martin Ellis, Yasaman Hosseinkashi, Sam Johnson, Ross Cutler

摘要:

用户体验质量(QoE)调查是重要的信息来源。除了用于估计平均意见得分(MOS)的常见“星级”之外,关于特定领域的更详细的调查问题(问题标记)提供了对影响QoE的因素的有价值的见解。本文从两个方面探讨了问题表征问卷的设计。首先,我们研究了固定问题顺序引入的偏差,其次,我们研究了选择问题子集以保持令牌集较小的挑战。基于从实时系统中使用随机控制实验收集的900,000个呼叫,我们发现通过随机化令牌的显示顺序可以显着降低顺序偏差。响应率的差异因令牌位置和显示设计而异。值得注意的是,用户对随机顺序变量的响应水平与固定顺序变量相当。令牌问题的子集的有效选择是通过提取令牌,提供最高的信息增益超过用户评级。这种选择被认为是NP难问题。我们在数据集上应用了一种著名的贪婪子模最大化方法,仅用30%的问题就捕获了94%的信息。

英文摘要:

User surveys for Quality of Experience (QoE) are a critical source of information. In addition to the common "star rating" used to estimate Mean Opinion Score (MOS), more detailed survey questions (problem tokens) about specific areas provide valuable insight into the factors impacting QoE. This paper explores two aspects of the problem token questionnaire design. First, we study the bias introduced by fixed question order, and second, we study the challenge of selecting a subset of questions to keep the token set small. Based on 900,000 calls gathered using a randomized controlled experiment from a live system, we find that the order bias can be significantly reduced by randomizing the display order of tokens. The difference in response rate varies based on token position and display design. It is worth noting that the users respond to the randomized-order variant at levels that are comparable to the fixed-order variant. The effective selection of a subset of token questions is achieved by extracting tokens that provide the highest information gain over user ratings. This selection is known to be in the class of NP-hard problems. We apply a well-known greedy submodular maximization method on our dataset to capture 94% of the information using just 30% of the questions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑