arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.04683cs.SD
收藏

UniAudio 2.0:一种具有文本对齐因子化音频标记化的统一音频语言模型

UniAudio 2.0: A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization

The Chinese University of Hong Kong, China(香港中文大学)

Dongchao Yang, Yuanyuan Wang, Dading Chong, Songxiang Liu, Xixin Wu, Helen Meng

AI总结:

UniAudio 2.0通过引入ReasoningCodec和统一自回归架构,实现了文本对齐的因子化音频标记化,提升了音频理解和生成性能,并在多种任务中展现了强大的泛化能力。

摘要:

我们研究了音频语言模型中的两个基础问题:(1)如何设计一种音频标记器,使其可以作为理解和生成的中间表示;(2)如何构建一个能够实现少样本和零样本泛化的音频基础模型,类似于大型语言模型。为此,我们做出了以下两个贡献。首先,我们提出了ReasoningCodec,一种离散音频编码器,将音频分解为(i)推理标记,这些标记编码了与文本对齐的高层分析和规划表示,用于音频理解和分层生成,以及(ii)重建标记,这些标记编码了语义丰富的声学线索,用于高保真的波形重建。这种设计在理解性能上与强大的连续表示相当,同时在生成质量和重建保真度方面优于之前的离散标记器。其次,我们引入了一种用于文本和音频的统一自回归架构,以及多阶段训练和多任务数据构建。使用此框架,我们在1000亿个文本标记和600亿个音频标记上训练了UniAudio 2.0。在广泛的语音、声音和音乐任务中,UniAudio 2.0在领域内评估中表现竞争,同时在未见任务上展示了强大的少样本和零样本泛化能力。演示、代码和检查点将在https://dongchaoyang.top/UniAudio2Demo/上提供。

英文摘要:

We study two foundational problems in audio language models: (1) how to design an audio tokenizer that can serve as an intermediate representation for both understanding and generation; and (2) how to build an audio foundation model that generalizes in few-shot and zero-shot settings, analogous to large language models. To this end, we make the following two contributions. First, we propose ReasoningCodec, a discrete audio codec that factorizes audio into (i) reasoning tokens, which encode text-aligned, high-level analysis and planning representations for audio understanding and hierarchical generation, and (ii) reconstruction tokens, which encode semantic-rich acoustic cues for high-fidelity waveform reconstruction. This design achieves understanding performance comparable to strong continuous representations while improving generation quality and reconstruction fidelity over prior discrete tokenizers. Second, we introduce a unified autoregressive architecture for text and audio, together with multi-stage training and multi-task data construction. Using this framework, we train UniAudio 2.0 on 100B text tokens and 60B audio tokens. Across a wide range of speech, sound, and music tasks, UniAudio 2.0 performs competitively on in-domain evaluations and demonstrates strong few-shot and zero-shot generalization to unseen tasks. Demo, code, and checkpoints will be available at \href{https://dongchaoyang.top/UniAudio2Demo/}{https://dongchaoyang.top/UniAudio2Demo/}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑