arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.26372cs.SD
收藏

UniTok-Audio:基于离散编解码器 Token 生成式建模的统一音频生成框架

UniTok-Audio: A Unified Audio Generation Framework via Generative Modeling on Discrete Codec Tokens

Alibaba Group(阿里巴巴集团);TongYi Ai lab(通义实验室);Zhejiang University(浙江大学)

Chengwei Liu, Haoyin Yan, Shaofei Xue, Xiaotao Liang, Yinghao Liu, Zheng Xue, Gang Song, Boyang Zhou

AI总结:

针对音频生成任务碎片化导致的开发冗余和泛化受限问题,提出统一音频生成框架 UniTok-Audio,通过自回归生成离散 token、任务标识符统一多任务模式及双流编解码器,在五项音频任务上取得与SOTA媲美的性能。

摘要:

生成式建模近期在文本、图像和音频领域取得了显著成功,展现了强大的统一表示学习能力。然而,音频生成模型在音频质量和跨任务泛化能力方面仍面临挑战。这种碎片化导致了开发工作冗余、性能不一致以及可扩展性受限。为了解决这些问题,我们提出了 UniTok-Audio,一个用于统一音频生成任务的可扩展且可扩展的框架。具体而言:1)UniTok-Audio 提取条件的连续特征,并以自回归方式生成目标音频的离散 token;2)一个特殊的任务标识符 token 在单一框架内统一了多个任务的不同学习模式;3)开发了包含声学和语义分支的双流音频编解码器,用于高保真波形重建。实验结果表明,UniTok-Audio 在五个时间对齐任务上取得了与最先进的任务特定或多任务系统相媲美的竞争性能:语音恢复、目标说话人提取、语音分离、语音转换和语言查询的音频源分离。为了促进未来研究,我们将开源我们的代码库。我们工作的演示页面可以在这里找到:https://alibaba.github.io/unified-audio。

英文摘要:

Generative modeling has recently achieved remarkable success across text, image, and audio domains, demonstrating powerful capabilities for unified representation learning. However, audio generation models still face challenges in terms of audio quality and generalization ability across tasks. This fragmentation results in redundant development efforts, inconsistent performance, and limited extensibility. To address these issues, we propose \textbf{UniTok-Audio}, a scalable and extensible framework for unified audio generation tasks. Specifically, 1) UniTok-Audio extracts continuous feature of conditions to generates discrete tokens of target audio in an autoregressive manner; 2) a special task identifier token unifies different learning patterns of multiple tasks in a single framework; 3) a dual-stream audio codec involving acoustic and semantic branch is developed for high-fidelity waveform reconstruction. Experimental results demonstrate that UniTok-Audio achieves competitive performance in comparation with state-of-the-art task-specific or multi-task systems across five time-aligned tasks: speech restoration, target speaker extraction, speech separation, voice conversion, and language-queried audio source separation. To foster future research, we will open-source our codebase. The demo page of our work can be found here: https://alibaba.github.io/unified-audio.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑