基于非自回归生成与预训练的直接语音到语音翻译中的说话人信息保留
Preserving Speaker Information in Direct Speech-to-Speech Translation with Non-Autoregressive Generation and Pretraining
AI总结：
本文提出一种基于自监督预训练和特征融合策略的直接语音到语音翻译方法，在保留说话人特征的同时实现非自回归生成，相比SC-S2UT提升了BLEU分数和说话人相似度，且推理时间仅增加0.04秒。
摘要：
语音到语音翻译（S2ST）是指将一种语言的语音转换为另一种语言中语义等价的语音，从而促进不同语言说话者之间的交流。语音到离散单元翻译（S2UT）是端到端S2ST的主流方法，它解决了传统级联系统中常见的跨模块错误传播和推理速度慢等挑战。然而，由于离散单元主要捕获内容信息，传统的S2UT方法无法保留源语音中说话人特有的特征。我们之前的工作SC-S2UT引入了说话人适配器和单元到梅尔结构，从而能够保留说话人信息并进行非自回归语音生成。在此基础上，本研究提出了一种自监督预训练方法，以丰富说话人适配器和单元到梅尔结构所提取的信息。此外，我们研究了不同的特征融合策略，以进一步改善说话人特征与内容特征的整合。在CVSS-T数据集上针对ES-EN和FR-EN任务进行的实验表明，我们提出的方法相比SC-S2UT实现了1.14的BLEU分数提升，同时在MOS和说话人相似度方面也有显著增强。此外，我们的方法在翻译质量上达到了与传统S2UT相当的水平，每句话的推理时间仅增加了0.04秒，同时保持了较高的说话人相似度。这些结果验证了所提方法的有效性。
英文摘要：
Speech-to-Speech Translation (S2ST) refers to the conversion of speech in one language into semantically equivalent speech in another language, facilitating communication between speakers of different languages. Speech-to-Discrete Unit Translation (S2UT), a mainstream approach for end-to-end S2ST, addresses challenges such as error propagation across modules and slow inference speed often encountered in traditional cascade systems. However, as discrete units primarily capture content information, conventional S2UT methods fail to retain speaker-specific characteristics from the source. Our previous work, SC-S2UT, introduced a speaker adapter and a unit-to-mel structure, enabling the preservation of speaker information and non-autoregressive speech generation. Building on this foundation, this study proposes a self-supervised pretraining method to enrich the information extracted by both the speaker adapter and the unit-to-mel structure. Additionally, we investigate different feature fusion strategies to further improve the integration of speaker and content features. Experiments conducted on the CVSS-T dataset for ES-EN and FR-EN tasks demonstrate that our proposed method achieves a BLEU score improvement of 1.14 compared to SC-S2UT, along with significant enhancements in MOS and speaker similarity. Furthermore, our approach achieves translation quality comparable to traditional S2UT, with only a minimal increase of 0.04s per utterance in inference time, while maintaining high speaker similarity. These results validate the effectiveness of the proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
