仅使用单语语料库增强大语言模型的语码转换文本到语音合成能力
Enhancing Code-switched Text-to-Speech Synthesis Capability in Large Language Models with only Monolingual Corpora
AI总结：
本文提出CS-LLM，通过多语言语音任务和单语语料拼接构建语码转换数据，在仅用单语数据的情况下提升大语言模型的语码转换语音合成能力。
摘要：
尽管大语言模型（LLMs）在语音生成与识别方面已展现出潜力，但其应用主要局限于单语场景，在语码转换（CS）情境中的探索十分有限。本文提出了一种语码转换大语言模型（CS-LLM），旨在仅利用单语语料库增强LLMs的语码转换文本到语音合成（CS TTS）能力。具体而言，我们首先通过多语言语音识别与合成任务增强LLMs的多语言语音处理能力。随后，我们开发了一种有效的语码转换（CS）数据构建策略，该策略从不同的单语语音语料库中拆分并拼接单词，从而使LLMs具备更强的CS TTS能力。实验表明，即使在数据有限的情况下，我们的方法在CS TTS的自然度、说话人一致性和相似度方面均优于基线方法。此外，所构建的CS数据还进一步提升了多语言语音合成与识别的性能。
英文摘要：
While Large Language Models (LLMs) have shown potential in speech generation and recognition, their applications are mainly confined to monolingual scenarios, with limited explorations in code-switched (CS) contexts. In this paper, we propose a Code-Switched Large Language Model (CS-LLM) to enhance the code-switched text-to-speech synthesis (CS TTS) capability in LLMs with only monolingual corpora. Specifically, we begin by enhancing the multilingual speech processing ability of LLMs through multilingual speech recognition and synthesis tasks. Then, we develop an effective code-switched (CS) data construction strategy that splits and concatenates words from different monolingual speech corpora to equip LLMs with improved CS TTS ability. Experiments show that our approach outperforms baselines in CS TTS in terms of naturalness, speaker consistency and similarity even with limited data. Additionally, the constructed CS data further improves multilingual speech synthesis and recognition.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
