arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.14294cs.SDcs.AIeess.AS
收藏

DASB - 离散音频与语音基准

DASB - Discrete Audio and Speech Benchmark

Concordia University(康科迪亚大学);Mila-Quebec AI Institute(蒙特利尔AI研究院);Avignon Université(阿维尼昂大学);Université de Montréal(蒙特利尔大学);Université Laval(拉瓦尔大学);Carnegie Mellon University(卡内基梅隆大学);Microsoft(微软);Indiana University(印第安纳大学)

Pooneh Mousavi, Jarod Duret, Darius Petermann, Artem Ploujnikov, Luca Della Libera, Anastasia Kuznetsova, Cem Subakan, Mirco Ravanelli

AI总结:

本文提出DASB基准,用于评估离散音频token在语音、通用音频和音乐领域的表现,发现离散表示不如连续表示鲁棒,需精细调参,语义token优于声学token,但与连续特征仍有差距。

摘要:

离散音频token近年来因其在连接音频与语言处理中的潜力而受到关注,能够使多模态语言模型既能生成又能理解音频。然而,保持如语音内容、说话人身份和副语言线索等关键信息仍是一个主要挑战。识别最优分词器和配置进一步受到现有研究中评估设置不一致的困扰。为此,我们引入了离散音频与语音基准(DASB),这是一个全面的框架,用于在语音、通用音频和音乐领域的一系列判别和生成任务中评估离散音频token。我们的结果表明,离散表示比连续表示更不鲁棒,需要仔细调整模型架构、数据量、学习率和容量等因素。语义token通常优于声学token,但离散token与连续特征之间仍存在差距,凸显了进一步研究的必要性。DASB代码、评估设置和排行榜可在https://poonehmousavi.github.io/DASB-website/上公开获取。

英文摘要:

Discrete audio tokens have recently gained considerable attention for their potential to bridge audio and language processing, enabling multimodal language models that can both generate and understand audio. However, preserving key information such as phonetic content, speaker identity, and paralinguistic cues remains a major challenge. Identifying the optimal tokenizer and configuration is further complicated by inconsistent evaluation settings across existing studies. To address this, we introduce the Discrete Audio and Speech Benchmark (DASB), a comprehensive framework for benchmarking discrete audio tokens across speech, general audio, and music domains on a range of discriminative and generative tasks. Our results show that discrete representations are less robust than continuous ones and require careful tuning of factors such as model architecture, data size, learning rate, and capacity. Semantic tokens generally outperform acoustic tokens, but a gap remains between discrete tokens and continuous features, highlighting the need for further research. DASB codes, evaluation setup, and leaderboards are publicly available at https://poonehmousavi.github.io/DASB-website/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑