HoliTok: 一种具有鲁棒的双重语音生成与理解能力的连续整体式分词
HoliTok: A Continuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding
AI总结:
提出HoliTok连续整体式语音分词模型,通过渐进训练策略联合保持信号保真度、融入语义信息并维持潜在可学习性,基于该分词构建统一AR+DiT模型实现语音合成与识别,实验证明其在统一生成-理解架构中无需额外优化即可鲁棒运行。
摘要:
统一的语音基础模型需要一个整体式的分词空间,该空间既要能被语言模型学习,又要能解码为高质量波形。然而,现有的语音分词器往往无法同时满足这些要求,导致架构复杂度和训练设计增加。我们提出HoliTok,一种用于统一生成-理解建模的连续整体式语音分词模型。HoliTok将48 kHz语音编码为紧凑的25 Hz序列,包含128维潜在向量。它采用渐进策略进行训练,联合保留信号级保真度、融入语义信息并保持强大的潜在可学习性。基于此分词,我们构建了一个统一的AR+DiT模型用于语音合成和识别,其中相同的潜在序列既支持生成特定任务,也支持统一的生成-理解任务。实验表明,HoliTok实现了有竞争力的重建保真度,提高了高质量和可控合成中的生成可学习性,并且在评估的表示中,它是唯一一个在我们的统一生成-理解架构中无需额外优化技巧即可鲁棒运行的表示。这些结果表明,HoliTok作为一种有效的语音分词器,为统一口语建模提供了基础的表示接口。代码可在 https://github.com/bovod-sjtu/HoliTok 获取。
英文摘要:
Unified speech foundation models require a holistic tokenization space that is both learnable by language models and decodable into high-quality waveforms. Existing speech tokenizers, however, often fail to satisfy these requirements simultaneously, leading to increased architectural complexity and more involved training designs. We propose HoliTok, a continuous Holistic speech Tokenization model designed for unified generation-understanding modeling. HoliTok encodes 48~kHz speech into a compact 25~Hz sequence of 128-dimensional latents. It is trained with a progressive strategy that jointly preserves signal-level fidelity, incorporates semantic information, and maintains strong latent learnability. Based on this tokenization, we build a unified AR+DiT model for speech synthesis and recognition, where the same latent sequence supports both generation-specific and unified generation-understanding tasks. Experiments show that HoliTok achieves competitive reconstruction fidelity, improves generative learnability for high-quality and controllable synthesis, and, among the evaluated representations, is the only one that operates robustly in our unified generation-understanding architecture without additional optimization tricks. These results suggest that HoliTok serves as an effective speech tokenizer and a foundational representation interface for unified spoken language modeling. The code is available at: https://github.com/bovod-sjtu/HoliTok.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。