可再现的主观评价
Reproducible Subjective Evaluation
摘要：
人类感知研究是评估机器学习、语言学和心理学中许多研究任务的黄金标准。然而，这些研究需要大量的时间和成本来执行。因此，许多研究人员使用的客观指标与人类评估的相关性很差。在进行主观评价时，往往没有提供足够的细节来确保可重复性。我们提出了ReSEval（Reproducible Subjective Evaluation），这是一个开源框架，用于直接从Python快速部署众包主观评估。ReSEval允许研究人员从命令行界面或使用一行Python对音频，图像，文本或视频数据进行A/B，ABX，平均意见评分（MOS）和具有隐藏参考和锚点的多刺激（MUSHRA）测试，使其与客观评估一样容易运行。使用ReSEval，研究人员可以通过共享配置文件和音频、图像、文本或视频文件来重现彼此的主观评估。
英文摘要：
Human perceptual studies are the gold standard for the evaluation of many research tasks in machine learning, linguistics, and psychology. However, these studies require significant time and cost to perform. As a result, many researchers use objective measures that can correlate poorly with human evaluation. When subjective evaluations are performed, they are often not reported with sufficient detail to ensure reproducibility. We propose Reproducible Subjective Evaluation (ReSEval), an open-source framework for quickly deploying crowdsourced subjective evaluations directly from Python. ReSEval lets researchers launch A/B, ABX, Mean Opinion Score (MOS) and MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA) tests on audio, image, text, or video data from a command-line interface or using one line of Python, making it as easy to run as objective evaluation. With ReSEval, researchers can reproduce each other's subjective evaluations by sharing a configuration file and the audio, image, text, or video files.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
