VQ-CTAP:面向语音处理的跨模态细粒度序列表示学习
VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing
AI总结:
提出VQ-CTAP跨模态序列表示学习范式,通过向量量化对比预训练和序列转换器实现帧级文本与语音对齐,无需微调即可应用于VC和ASR,并通过序列感知连接器支持TTS,实现960倍高压缩语音编码。
摘要:
深度学习为跨模态表示学习领域带来了显著改进。对于文本到语音(TTS)、语音转换(VC)和自动语音识别(ASR)等任务,需要一种跨模态细粒度(帧级)序列表示,这种表示强调文本模态的语义内容,同时弱化语音模态的副语言信息。我们提出了一种名为“向量量化对比Token-声学预训练(VQ-CTAP)”的方法,该方法使用跨模态对齐的序列转换器将文本和语音引入联合多模态空间,学习如何在帧级别上连接文本和语音。提出的VQ-CTAP是一种跨模态序列表示学习的范式,为语音处理中的细粒度生成和识别任务提供了有前景的解决方案。VQ-CTAP可直接应用于VC和ASR任务,无需微调或额外结构。我们提出了一种序列感知语义连接器,它为TTS任务连接多个冻结的预训练模块,展现出即插即用的能力。我们设计了步进优化策略,通过逐步注入和调整各损失成分的影响来确保模型有效收敛。此外,我们提出了语义转移副语言一致性损失以增强表示能力,使模型能更好地泛化到未见数据并捕捉副语言信息的细微差别。另外,VQ-CTAP实现了从24kHz输入波形到25Hz的高压缩语音编码,采样率降低了960倍。音频演示请见 https://qiangchunyu.github.io/VQCTAP/
英文摘要:
Deep learning has brought significant improvements to the field of cross-modal representation learning. For tasks such as text-to-speech (TTS), voice conversion (VC), and automatic speech recognition (ASR), a cross-modal fine-grained (frame-level) sequence representation is desired, emphasizing the semantic content of the text modality while de-emphasizing the paralinguistic information of the speech modality. We propose a method called "Vector Quantized Contrastive Token-Acoustic Pre-training (VQ-CTAP)", which uses the cross-modal aligned sequence transcoder to bring text and speech into a joint multimodal space, learning how to connect text and speech at the frame level. The proposed VQ-CTAP is a paradigm for cross-modal sequence representation learning, offering a promising solution for fine-grained generation and recognition tasks in speech processing. The VQ-CTAP can be directly applied to VC and ASR tasks without fine-tuning or additional structures. We propose a sequence-aware semantic connector, which connects multiple frozen pre-trained modules for the TTS task, exhibiting a plug-and-play capability. We design a stepping optimization strategy to ensure effective model convergence by gradually injecting and adjusting the influence of various loss components. Furthermore, we propose a semantic-transfer-wise paralinguistic consistency loss to enhance representational capabilities, allowing the model to better generalize to unseen data and capture the nuances of paralinguistic information. In addition, VQ-CTAP achieves high-compression speech coding at a rate of 25Hz from 24kHz input waveforms, which is a 960-fold reduction in the sampling rate. The audio demo is available at https://qiangchunyu.github.io/VQCTAP/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。