arXiv 2409.158972025-02-24eess.AScs.SD

ESPnet-Codec：音频、音乐和语音神经编解码器的综合训练与评估

ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech

Carnegie Mellon University（卡内基梅隆大学）；Renmin University of China（中国人民大学）；Nanyang Technological University（南洋理工大学）；Tokyo Metropolitan University（东京都立大学）；Fudan University（复旦大学）；University of Chicago（芝加哥大学）；National Taiwan University（台湾大学）；Massachusetts Institute of Technology（麻省理工学院）

Jiatong Shi, Jinchuan Tian, Yihan Wu, Jee-weon Jung, Jia Qi Yip, Yoshiki Masuyama, William Chen, Yuning Wu, Yuxun Tang, Massa Baali, Dareen Alharhi, Dong Zhang,… 展开作者

Jiatong Shi, Jinchuan Tian, Yihan Wu, Jee-weon Jung, Jia Qi Yip, Yoshiki Masuyama, William Chen, Yuning Wu, Yuxun Tang, Massa Baali, Dareen Alharhi, Dong Zhang, Ruifan Deng, Tejes Srivastava, Haibin Wu, Alexander H. Liu, Bhiksha Raj, Qin Jin, Ruihua Song, Shinji Watanabe

备注：: Accepted by SLT

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出ESPnet-Codec开源平台，基于ESPnet支持音频、音乐和语音神经编解码器的训练与评估，并配套VERSA工具包，通过20个指标全面评估，可集成至六个ESPnet任务。

摘要：

神经编解码器已成为近期语音和音频生成研究的关键。除了信号压缩能力外，离散编解码器还被发现能提升下游训练效率，并与自回归语言模型兼容。然而，随着大量下游应用被研究，确保不同应用间公平比较的挑战也随之出现。为解决这些问题，我们提出了一个新的开源平台ESPnet-Codec，它基于ESPnet构建，专注于神经编解码器的训练和评估。ESPnet-Codec提供了音频、音乐和语音领域的多种配方，用于使用多种广泛采用的编解码器模型进行训练和评估。与ESPnet-Codec一同，我们提出了VERSA，一个独立的评估工具包，它通过20个音频评估指标对编解码器性能进行全面评估。值得注意的是，我们证明了ESPnet-Codec可以集成到六个ESPnet任务中，支持多种应用。

英文摘要：

Neural codecs have become crucial to recent speech and audio generation research. In addition to signal compression capabilities, discrete codecs have also been found to enhance downstream training efficiency and compatibility with autoregressive language models. However, as extensive downstream applications are investigated, challenges have arisen in ensuring fair comparisons across diverse applications. To address these issues, we present a new open-source platform ESPnet-Codec, which is built on ESPnet and focuses on neural codec training and evaluation. ESPnet-Codec offers various recipes in audio, music, and speech for training and evaluation using several widely adopted codec models. Together with ESPnet-Codec, we present VERSA, a standalone evaluation toolkit, which provides a comprehensive evaluation of codec performance over 20 audio evaluation metrics. Notably, we demonstrate that ESPnet-Codec can be integrated into six ESPnet tasks, supporting diverse applications.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。