arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.10934cs.SDeess.AS
收藏

MOSS-Audio-Tokenizer: 为未来音频基础模型扩展音频分词器

MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models

Yitian Gong, Kuangwei Chen, Zhaoye Fei, Xiaogui Yang, Ke Chen, Yang Wang, Kexin Huang, Mingshu Chen, Ruixiao Li, Qingyuan Cheng, Shimin Li, Xipeng Qiu

AI总结:

MOSS-Audio-Tokenizer通过同质Transformer架构实现端到端音频分词,支持高保真重建并提升语音合成和语音识别性能。

摘要:

离散的音频分词器是使大型语言模型具备原生音频处理和生成能力的关键。尽管近年来取得了一定进展,但现有方法往往依赖于预训练编码器、语义蒸馏或异构CNN架构。这些设计引入了固定的归纳偏差,限制了重建保真度并阻碍了有效扩展。在本文中,我们主张通过一个同质且可扩展的架构,完全端到端地学习离散音频分词。为此,我们首先提出CAT(带有Transformer的因果音频分词器),这是一个纯基于Transformer的架构,从头开始联合优化编码器、量化器和解码器,以实现高保真重建。基于CAT架构,我们开发了MOSS-Audio-Tokenizer,一个具有160亿参数的大型音频分词器,预训练于300万小时的多样化通用音频数据。我们展示了这种简单、完全端到端的方法,通过同质、因果的Transformer块,能够优雅地扩展并支持在不同音频领域中的高保真重建。在语音、声音和音乐方面,MOSS-Audio-Tokenizer在各种比特率范围内均优于先前的编解码器,同时随着规模的增加,表现出可预测的改进。值得注意的是,利用我们模型的离散令牌,我们开发了第一个纯自回归TTS模型,其性能超越了先前的非自回归和级联系统。此外,MOSS-Audio-Tokenizer在不使用辅助编码器的情况下实现了具有竞争力的ASR性能。我们的发现将CAT架构定位为下一代原生音频基础模型的统一、可扩展的接口。

英文摘要:

Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretrained encoders, semantic distillation, or heterogeneous CNN-based architectures. These designs introduce fixed inductive biases that limit reconstruction fidelity and hinder effective scaling. In this paper, we argue that discrete audio tokenization should be learned fully end-to-end using a homogeneous and scalable architecture. To this end, we first propose CAT (Causal Audio Tokenizer with Transformer), a purely Transformer-based architecture that jointly optimizes the encoder, quantizer, and decoder from scratch for high-fidelity reconstruction. Building on the CAT architecture, we develop MOSS-Audio-Tokenizer, a large-scale audio tokenizer featuring 1.6 billion parameters, pre-trained on 3 million hours of diverse, general audio data. We show that this simple, fully end-to-end approach built from homogeneous, causal Transformer blocks scales gracefully and supports high-fidelity reconstruction across diverse audio domains. Across speech, sound, and music, MOSS-Audio-Tokenizer consistently outperforms prior codecs over a wide range of bitrates, while exhibiting predictable improvements with increased scale. Notably, leveraging the discrete tokens from our model, we develop the first purely autoregressive TTS model that surpasses prior non-autoregressive and cascaded systems. Furthermore, MOSS-Audio-Tokenizer enables competitive ASR performance without auxiliary encoders. Our findings position the CAT architecture as a unified, scalable interface for the next generation of native audio foundation models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑