研究大规模伪立体声数据与不同语音基础模型对对话生成式口语模型的影响
Investigating the Effects of Large-Scale Pseudo-Stereo Data and Different Speech Foundation Model on Dialogue Generative Spoken Language Model
AI总结：
本文提出将单声道对话转换为伪立体声数据的流程，把训练数据扩展至17,600小时，并比较不同语音基础模型的离散单元，以提升口语对话生成模型性能。
摘要：
近期，Spoken Dialogue Modeling（口语对话建模）领域的研究致力于在无需直接转写的情况下合成口语对话，从而保留语音中固有的丰富非文本信息。然而，当说话者同时讲话时，该方法面临挑战，需要使用将不同说话者录制在独立声道上的立体声对话数据，而这一资源相当稀缺。为解决这一问题，我们开发了一种创新流程，能够将单声道对话数据转换为伪立体声数据。这使我们的训练数据集从仅有2,000小时扩展到令人瞩目的17,600小时，显著丰富了可用训练样本的多样性和质量。事实证明，纳入这些伪立体声数据可有效提升口语对话语言模型的性能。此外，我们还探索了将不同语音基础模型的离散单元用于口语对话生成。
英文摘要：
Recent efforts in Spoken Dialogue Modeling aim to synthesize spoken dialogue without the need for direct transcription, thereby preserving the wealth of non-textual information inherent in speech. However, this approach faces a challenge when speakers talk simultaneously, requiring stereo dialogue data with speakers recorded on separate channels, a notably scarce resource. To address this, we have developed an innovative pipeline capable of transforming single-channel dialogue data into pseudo-stereo data. This expanded our training dataset from a mere 2,000 to an impressive 17,600 hours, significantly enriching the diversity and quality of the training examples available. The inclusion of this pseudo-stereo data has proven to be effective in improving the performance of spoken dialogue language models. Additionally, we explored the use of discrete units of different speech foundation models for spoken dialogue generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
