arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.06407eess.AScs.AIcs.CL
收藏

WavCube:通过语义-声音联合建模统一语音表示以实现理解和生成

WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling

Shanghai Jiao Tong University(上海交通大学);Shanghai Innovation Institute(上海创新研究院);Tencent(腾讯);Independent Researcher(独立研究员);Peking University(北京大学);Tianjin University(天津大学);Zhejiang University(浙江大学)

Guanrou Yang, Tian Tan, Qian Chen, Zhikang Niu, Yakun Song, Ziyang Ma, Yushen Chen, Zeyu Xie, Tianrui Wang, Yifan Yang, Wenxi Chen, Qi Chen, Wenrui Liu, Shan Yang, Xie Chen

AI总结:

WavCube通过语义-声音联合建模统一语音表示,解决语音理解和生成的表示兼容问题,实现压缩后的高性能语音任务表现。

摘要:

将语音理解和生成整合是构建统一语音模型的关键步骤。然而,这两个任务所需的不同表示目前存在显著的兼容性挑战。通常,语义导向的特征通过自监督学习(SSL)学习,而声音导向的特征通过重建学习。这种碎片化的表示阻碍了真正统一的语音系统的实现。我们提出了WavCube,一种从SSL语音编码器中衍生出的紧凑连续潜在表示,能够同时支持语音理解、重建和生成。WavCube采用两阶段训练方案。第一阶段训练一个语义瓶颈,过滤掉使原始SSL特征难以处理的冗余信息。第二阶段通过端到端重建注入细粒度的声音细节,同时语义锚定损失确保表示始终保持在原始语义流形上。全面的实验表明,尽管在维度上压缩了8倍,WavCube在SUPERB上接近WavLM的性能,重建质量与现有声音表示相当,实现了最先进的零样本TTS性能,训练收敛速度显著加快,并在SUPERB-SG基准测试中在语音增强、分离和语音转换任务中表现出色。系统消融研究揭示,WavCube的两阶段配方解决了SSL特征在生成建模中的两个内在缺陷,为未来统一的语音系统铺平了道路。代码和检查点可在https://github.com/yanghaha0908/WavCube上获取。

英文摘要:

Integrating speech understanding and generation is a pivotal step toward building unified speech models. However, the different representations required for these two tasks currently pose significant compatibility challenges. Typically, semantics-oriented features are learned from self-supervised learning (SSL), and acoustic-oriented features from reconstruction. Such fragmented representations hinder the realization of truly unified speech systems. We present WavCube, a compact continuous latent derived from an SSL speech encoder that simultaneously supports speech understanding, reconstruction, and generation. WavCube employs a two-stage training scheme. Stage 1 trains a semantic bottleneck to filter off-manifold redundancy that makes raw SSL features intractable for diffusion. Stage 2 injects fine-grained acoustic details via end-to-end reconstruction, while a semantic anchoring loss ensures the representation remains grounded within its original semantic manifold. Comprehensive experiments show that WavCube closely approaches WavLM performance on SUPERB despite an 8x dimensional compression, attains reconstruction quality on par with existing acoustic representations, delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence, and excels in speech enhancement, separation, and voice conversion tasks on the SUPERB-SG benchmark. Systematic ablations reveal that WavCube's two-stage recipe resolves two intrinsic flaws of SSL features for generative modeling, paving the way for future unified speech systems. Codes and checkpoints are available at https://github.com/yanghaha0908/WavCube.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑