arXiv 2407.072752024-08-26eess.AScs.CLcs.LGcs.SD

重塑 Divide and Remaster：一个支持多语言的影视音频源分离数据集

Remastering Divide and Remaster: A Cinematic Audio Source Separation Dataset with Multilingual Support

Netflix, Inc.（网飞公司）；Georgia Institute of Technology（佐治亚理工学院）

Karn N. Watcharasupat, Chih-Wei Wu, Iroro Orife

备注：: Accepted to the 5th IEEE International Symposium on the Internet of Sounds. Camera-ready version

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出了DnR v3数据集，通过改进非对白人声、响度分布、母带制作并引入超30种语言支持，解决了影视音频源分离任务中现有数据集的局限性，实验证明多语言训练显著提升了模型的泛化能力。

摘要：

影视音频源分离（CASS）作为从混合音频中提取对白、音乐和音效分轨的问题，是音频源分离的一个相对较新的子任务。迄今为止，CASS仅存在一个公开可用的数据集，即Divide and Remaster（DnR）数据集，目前版本为v2。尽管DnR v2一直是CASS极其有用的资源，但通过其在2023年Sound Demixing Challenge中的应用，已发现其存在几个有待改进的领域，特别是在非对白分轨的人声内容、响度分布、母带制作过程以及语言多样性方面。在本研究中，我们开发了DnR数据集的版本3，解决了与非对白分轨中的人声内容、响度分布、母带制作过程和语言多样性相关的问题。特别是，DnR v3的对白分轨包含来自多个语系超过30种语言的语音内容，包括但不限于日耳曼语族、罗曼语族、印度-雅利安语族、达罗毗荼语族、马来-波利尼西亚语族和班图语族。使用Bandit模型的基准测试结果表明，在多语言数据上进行训练能为模型带来显著的泛化能力，即使在数据可用性较低的语言中也是如此。即使在数据可用性较高的语言中，多语言模型的表现通常也与应用在单语CASS数据集上训练的专用模型相当或更好。数据集和模型实现将在https://github.com/kwatcharasupat/source-separation-landing上提供。

英文摘要：

Cinematic audio source separation (CASS), as a problem of extracting the dialogue, music, and effects stems from their mixture, is a relatively new subtask of audio source separation. To date, only one publicly available dataset exists for CASS, that is, the Divide and Remaster (DnR) dataset, which is currently at version 2. While DnR v2 has been an incredibly useful resource for CASS, several areas of improvement have been identified, particularly through its use in the 2023 Sound Demixing Challenge. In this work, we develop version 3 of the DnR dataset, addressing issues relating to vocal content in non-dialogue stems, loudness distributions, mastering process, and linguistic diversity. In particular, the dialogue stem of DnR v3 includes speech content from more than 30 languages from multiple families including but not limited to the Germanic, Romance, Indo-Aryan, Dravidian, Malayo-Polynesian, and Bantu families. Benchmark results using the Bandit model indicated that training on multilingual data yields significant generalizability to the model even in languages with low data availability. Even in languages with high data availability, the multilingual model often performs on par or better than dedicated models trained on monolingual CASS datasets. Dataset and model implementation will be made available at https://github.com/kwatcharasupat/source-separation-landing.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。