FocalCodec：基于焦点调制网络的低比特率语音编码
FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks
AI总结：
本文提出FocalCodec，一种基于焦点调制的低比特率语音编解码器，利用单一二进制码本在0.16-0.65 kbps下压缩语音，兼顾语义与声学信息，性能优于现有方法，并支持多语言及噪声环境。
摘要：
大型语言模型通过对海量数据集进行自监督预训练，彻底改变了自然语言处理领域。受此成功启发，研究人员探索了将这些方法应用于语音，通过使用神经音频编解码器将连续音频离散化为标记。然而，现有方法面临诸多限制，包括高比特率、语义或声学信息的丢失，以及在试图同时捕获两者时对多码本设计的依赖，这增加了下游任务的架构复杂性。为了解决这些挑战，我们引入了FocalCodec，一种基于焦点调制的高效低比特率编解码器，它利用单个二进制码本将语音压缩至0.16至0.65 kbps。FocalCodec在语音重建和语音转换方面，以低于当前最先进技术的比特率提供了具有竞争力的性能，同时有效处理多语言语音和嘈杂环境。对下游任务的评估表明，FocalCodec成功保留了足够的语义和声学信息，同时也非常适合生成建模。演示样本和代码可在https://lucadellalib.github.io/focalcodec-web/获取。
英文摘要：
Large language models have revolutionized natural language processing through self-supervised pretraining on massive datasets. Inspired by this success, researchers have explored adapting these methods to speech by discretizing continuous audio into tokens using neural audio codecs. However, existing approaches face limitations, including high bitrates, the loss of either semantic or acoustic information, and the reliance on multi-codebook designs when trying to capture both, which increases architectural complexity for downstream tasks. To address these challenges, we introduce FocalCodec, an efficient low-bitrate codec based on focal modulation that utilizes a single binary codebook to compress speech between 0.16 and 0.65 kbps. FocalCodec delivers competitive performance in speech resynthesis and voice conversion at lower bitrates than the current state-of-the-art, while effectively handling multilingual speech and noisy environments. Evaluation on downstream tasks shows that FocalCodec successfully preserves sufficient semantic and acoustic information, while also being well-suited for generative modeling. Demo samples and code are available at https://lucadellalib.github.io/focalcodec-web/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
