arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.06358cs.SDcs.LGeess.AS
收藏

使用评分员和系统元数据解释2022年VoiceMOS挑战赛数据集中的差异

Using Rater and System Metadata to Explain Variance in the VoiceMOS Challenge 2022 Dataset

Google(谷歌);University College Dublin(都柏林大学学院)

Michael Chinen, Jan Skoglund, Chandan K A Reddy, Alessandro Ragano, Andrew Hines

摘要:

非参考语音质量模型对于越来越多的应用是重要的。VoiceMOS 2022挑战赛提供了一个合成语音转换和带有主观标签的文本到语音样本的数据集。这项研究着眼于可以解释的语音质量的主观评级从元数据和数据集的分布不平衡的方差量。语音质量模型使用wav2vec 2.0构建,具有额外的元数据特征,包括评分员组和系统标识符,并获得了竞争性指标,包括斯皮尔曼等级相关系数(SRCC)为0.934和MSE为0.088,在系统级,在话语级为0.877和0.198。使用测试限制或盲化的数据和元数据进一步改进了度量。元数据分析表明,系统级指标并不代表模型的系统级预测,因为验证和测试数据集上每个系统使用的话语数量变化很大。我们的结论是,在一般情况下,条件应该有足够的话语在测试集约束样本的平均误差,并在系统之间的话语计数相对平衡,否则话语水平的度量可能更可靠和可解释的。

英文摘要:

Non-reference speech quality models are important for a growing number of applications. The VoiceMOS 2022 challenge provided a dataset of synthetic voice conversion and text-to-speech samples with subjective labels. This study looks at the amount of variance that can be explained in subjective ratings of speech quality from metadata and the distribution imbalances of the dataset. Speech quality models were constructed using wav2vec 2.0 with additional metadata features that included rater groups and system identifiers and obtained competitive metrics including a Spearman rank correlation coefficient (SRCC) of 0.934 and MSE of 0.088 at the system-level, and 0.877 and 0.198 at the utterance-level. Using data and metadata that the test restricted or blinded further improved the metrics. A metadata analysis showed that the system-level metrics do not represent the model's system-level prediction as a result of the wide variation in the number of utterances used for each system on the validation and test datasets. We conclude that, in general, conditions should have enough utterances in the test set to bound the sample mean error, and be relatively balanced in utterance count between systems, otherwise the utterance-level metrics may be more reliable and interpretable.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑