Style-Talker：微调音频语言模型与基于风格的文本转语音模型以实现快速口语对话生成
Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation
AI总结：
本文提出Style-Talker框架，通过微调音频LLM和基于风格的TTS模型，结合用户输入音频的转录与语音风格生成响应，在保持对话自然度和连贯性的同时将速度提升50%以上。
摘要：
大型语言模型（LLMs）的快速发展极大地推动了基于文本的聊天机器人的发展，展示了其进行连贯且上下文相关对话的能力。然而，将这些进步扩展到端到端的语音到语音对话机器人仍然是一个巨大的挑战，这主要由于需要大量的数据集和计算资源。在流水线中级联自动语音识别（ASR）、LLM和文本转语音（TTS）模型的传统方法虽然有效，但由于缺乏输入音频及其转录文本与输出音频之间的直接交互，因此存在韵律不自然的问题。这些系统还受到ASR过程在实时应用中固有延迟的限制。本文介绍了Style-Talker，这是一个创新框架，通过微调音频LLM和基于风格的TTS模型来实现快速口语对话生成。Style-Talker接收用户输入音频，并使用转录的聊天历史和语音风格来生成响应的说话风格和文本。随后，TTS模型合成语音，然后将其播放给用户。在播放响应语音的同时，输入语音经过ASR处理以提取转录和说话风格，作为后续对话轮次的上下文。这种新颖的流水线加速了传统的级联ASR-LLM-TTS系统，同时整合了来自输入语音的丰富副语言信息。我们的实验结果表明，Style-Talker在对话自然度和连贯性方面显著优于传统的级联和语音到语音基线，同时速度提高了50%以上。
英文摘要：
The rapid advancement of large language models (LLMs) has significantly propelled the development of text-based chatbots, demonstrating their capability to engage in coherent and contextually relevant dialogues. However, extending these advancements to enable end-to-end speech-to-speech conversation bots remains a formidable challenge, primarily due to the extensive dataset and computational resources required. The conventional approach of cascading automatic speech recognition (ASR), LLM, and text-to-speech (TTS) models in a pipeline, while effective, suffers from unnatural prosody because it lacks direct interactions between the input audio and its transcribed text and the output audio. These systems are also limited by their inherent latency from the ASR process for real-time applications. This paper introduces Style-Talker, an innovative framework that fine-tunes an audio LLM alongside a style-based TTS model for fast spoken dialog generation. Style-Talker takes user input audio and uses transcribed chat history and speech styles to generate both the speaking style and text for the response. Subsequently, the TTS model synthesizes the speech, which is then played back to the user. While the response speech is being played, the input speech undergoes ASR processing to extract the transcription and speaking style, serving as the context for the ensuing dialogue turn. This novel pipeline accelerates the traditional cascade ASR-LLM-TTS systems while integrating rich paralinguistic information from input speech. Our experimental results show that Style-Talker significantly outperforms the conventional cascade and speech-to-speech baselines in terms of both dialogue naturalness and coherence while being more than 50% faster.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
