用$p$ -值P-P图描述主观实验一致性
Describing Subjective Experiment Consistency by $p$-Value P-P Plot
摘要：
有些现象如果没有主观测试就无法衡量。然而，主观测试是一个复杂的问题，有许多影响因素。这些相互作用产生精确或不正确的结果。研究人员需要一种工具来分类主观实验的结果，无论是一致的或不一致的。这是必要的，以便决定是否将收集的分数视为高质量的地面实况数据。知道主观分数是否可信是得出有效结论和基于这些分数构建功能工具的关键（例如，评估多媒体材料的感知质量的算法）。我们提供了一个工具来分类主观实验（及其所有结果），无论是一致或不一致。此外，该工具识别具有不规则分数分布的刺激。该方法基于将主观分数视为来自离散广义分数分布（GSD）的随机变量。GSD与拟合优度的自举G检验相结合，可以构建可视化实验一致性的$p$值P-P图。该工具保护研究人员使用不一致的主观数据。通过这种方式，它确保他们得出的结论和他们构建的工具更加精确和可信。所提出的方法的工作原理在21个现实生活中的多媒体质量主观实验的实验设计描述完全符合预期。
英文摘要：
There are phenomena that cannot be measured without subjective testing. However, subjective testing is a complex issue with many influencing factors. These interplay to yield either precise or incorrect results. Researchers require a tool to classify results of subjective experiment as either consistent or inconsistent. This is necessary in order to decide whether to treat the gathered scores as quality ground truth data. Knowing if subjective scores can be trusted is key to drawing valid conclusions and building functional tools based on those scores (e.g., algorithms assessing the perceived quality of multimedia materials). We provide a tool to classify subjective experiment (and all its results) as either consistent or inconsistent. Additionally, the tool identifies stimuli having irregular score distribution. The approach is based on treating subjective scores as a random variable coming from the discrete Generalized Score Distribution (GSD). The GSD, in combination with a bootstrapped G-test of goodness-of-fit, allows to construct $p$-value P-P plot that visualizes experiment's consistency. The tool safeguards researchers from using inconsistent subjective data. In this way, it makes sure that conclusions they draw and tools they build are more precise and trustworthy. The proposed approach works in line with expectations drawn solely on experiment design descriptions of 21 real-life multimedia quality subjective experiments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
