arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.09239cs.SDcs.AIeess.AS
收藏

DSA-Tokenizer:基于流匹配层次化融合的解耦语义-声学分词器

DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion

Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系);AI Lab, Leibniz Research Center, Huawei(华为利比全中心人工智能实验室)

Hanlin Zhang, Daxin Tan, Dehua Tao, Xiao Chen, Haochen Tan, Yunhe Li, Yuchen Cao, Linqi Song

AI总结:

提出DSA-Tokenizer,通过ASR监督语义令牌和mel谱重建监督声学令牌实现解耦,并引入层次化流匹配解码器和联合重构-上下文修补训练策略,实现高保真重构和跨语句语音克隆。

摘要:

语音分词器是全离散语音大语言模型的关键构建模块。现有的分词器要么优先考虑语义编码,将语义内容与声学风格不可分离地融合,要么实现不完全的语义-声学解耦。为了实现更好的解耦,我们提出了DSA-Tokenizer,它通过不同的优化约束将语音显式解耦为离散的语义和声学令牌。具体来说,语义令牌由ASR监督以捕获语言内容,而声学令牌专注于mel谱重构以编码风格。我们进一步引入了层次化流匹配解码器和联合重构-上下文修补训练策略,使模型能够支持高保真重构和跨语句语音克隆。为了加速推理,我们蒸馏了DiT解码器,将推理采样步数减少到4步,并通过GAN微调提高合成质量。实验表明,DSA-Tokenizer提供了强大的语义-声学解耦、可靠的可控语音克隆以及低WER/CER的高效高保真生成。此外,我们的结果表明,解耦分词为下游大模型语音生成提供了更有效的接口。音频样本可在https://anonymous.4open.science/w/DSA_Tokenizer_demo/获取。

英文摘要:

Speech tokenizers are a key building block of fully discrete Speech LLMs. Existing tokenizers either prioritize semantic encoding, fuse semantic content with acoustic style inseparably, or achieve incomplete semantic-acoustic disentanglement. To achieve better disentanglement, we propose DSA-Tokenizer, which explicitly disentangles speech into discrete semantic and acoustic tokens via distinct optimization constraints. Specifically, semantic tokens are supervised by ASR to capture linguistic content, while acoustic tokens focus on mel-spectrograms restoration to encode style. We further introduce a hierarchical Flow Matching decoder and a joint reconstruction-context inpainting training strategy, allowing the model to support both high-fidelity reconstruction and cross-utterance voice clone. To speed up inference, we distill the dit decoder to 4-step inference and improve synthesis quality with GAN fine-tuning. Experiments demonstrate that DSA-Tokenizer provides strong semantic-acoustic disentanglement, reliable controllable voice cloning, and efficient high-fidelity generation with low WER/CER. Moreover, our results suggest that disentangled tokenization provides a more effective interface for downstream large-model speech generation. Audio samples are avaialble at https://anonymous.4open.science/w/DSA_Tokenizer_demo/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑