arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.29948cs.SDcs.AIeess.AS
收藏

HoliTok: 一种具有鲁棒的双重语音生成与理解能力的连续整体式分词

HoliTok: A Continuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding

X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, China(X-LANCE实验室,计算机科学学院,上海交通大学,中国);hi lab, Xiaohongshu Inc, China(hi实验室,小红书公司,中国)

Bohan Li, Shi Lian, Hankun Wang, Yiwei Guo, Yu Xi, Zhihan Li, Da Zheng, Colin Zhang, Kai Yu

AI总结:

提出HoliTok连续整体式语音分词模型,通过渐进训练策略联合保持信号保真度、融入语义信息并维持潜在可学习性,基于该分词构建统一AR+DiT模型实现语音合成与识别,实验证明其在统一生成-理解架构中无需额外优化即可鲁棒运行。

摘要:

统一的语音基础模型需要一个整体式的分词空间,该空间既要能被语言模型学习,又要能解码为高质量波形。然而,现有的语音分词器往往无法同时满足这些要求,导致架构复杂度和训练设计增加。我们提出HoliTok,一种用于统一生成-理解建模的连续整体式语音分词模型。HoliTok将48 kHz语音编码为紧凑的25 Hz序列,包含128维潜在向量。它采用渐进策略进行训练,联合保留信号级保真度、融入语义信息并保持强大的潜在可学习性。基于此分词,我们构建了一个统一的AR+DiT模型用于语音合成和识别,其中相同的潜在序列既支持生成特定任务,也支持统一的生成-理解任务。实验表明,HoliTok实现了有竞争力的重建保真度,提高了高质量和可控合成中的生成可学习性,并且在评估的表示中,它是唯一一个在我们的统一生成-理解架构中无需额外优化技巧即可鲁棒运行的表示。这些结果表明,HoliTok作为一种有效的语音分词器,为统一口语建模提供了基础的表示接口。代码可在 https://github.com/bovod-sjtu/HoliTok 获取。

英文摘要:

Unified speech foundation models require a holistic tokenization space that is both learnable by language models and decodable into high-quality waveforms. Existing speech tokenizers, however, often fail to satisfy these requirements simultaneously, leading to increased architectural complexity and more involved training designs. We propose HoliTok, a continuous Holistic speech Tokenization model designed for unified generation-understanding modeling. HoliTok encodes 48~kHz speech into a compact 25~Hz sequence of 128-dimensional latents. It is trained with a progressive strategy that jointly preserves signal-level fidelity, incorporates semantic information, and maintains strong latent learnability. Based on this tokenization, we build a unified AR+DiT model for speech synthesis and recognition, where the same latent sequence supports both generation-specific and unified generation-understanding tasks. Experiments show that HoliTok achieves competitive reconstruction fidelity, improves generative learnability for high-quality and controllable synthesis, and, among the evaluated representations, is the only one that operates robustly in our unified generation-understanding architecture without additional optimization tricks. These results suggest that HoliTok serves as an effective speech tokenizer and a foundational representation interface for unified spoken language modeling. The code is available at: https://github.com/bovod-sjtu/HoliTok.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑