IntrinsicVoice：赋予LLM内在实时语音交互能力
IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities
AI总结：
本文提出IntrinsicVoice，通过GroupFormer架构将语音序列压缩至与文本相当长度，并构建IntrinsicVoice-500k数据集与跨模态训练策略，实现LLM低于100ms延迟的实时语音交互。
摘要：
当前构建具备语音交互能力的LLM的方法，在生成语音响应之前或期间严重依赖显式的文本自回归生成来维持内容质量，这不幸地带来了计算开销，并增加了多轮交互中的延迟。为了解决这个问题，我们引入了IntrinsicVoice，一个设计有内在实时语音交互能力的LLM。IntrinsicVoice旨在通过缓解文本和语音之间的模态差距，促进预训练LLM的文本能力向语音模态的转移。我们的新颖架构GroupFormer，可以将语音序列缩减到与文本序列相当的长度，同时生成高质量音频，显著减少了语音和文本之间的长度差异，加快了推理速度，并缓解了长文本建模问题。此外，我们构建了一个名为IntrinsicVoice-500k的多轮语音到语音对话数据集，其中包含近50万轮语音到语音对话，以及一个跨模态训练策略，以增强语音和文本之间的语义对齐。实验结果表明，IntrinsicVoice在多轮对话场景中能够生成高质量语音响应，延迟低于100ms。演示可在https://instrinsicvoice.github.io/获取。
英文摘要：
Current methods of building LLMs with voice interaction capabilities rely heavily on explicit text autoregressive generation before or during speech response generation to maintain content quality, which unfortunately brings computational overhead and increases latency in multi-turn interactions. To address this, we introduce IntrinsicVoic,e an LLM designed with intrinsic real-time voice interaction capabilities. IntrinsicVoice aims to facilitate the transfer of textual capabilities of pre-trained LLMs to the speech modality by mitigating the modality gap between text and speech. Our novelty architecture, GroupFormer, can reduce speech sequences to lengths comparable to text sequences while generating high-quality audio, significantly reducing the length difference between speech and text, speeding up inference, and alleviating long-text modeling issues. Additionally, we construct a multi-turn speech-to-speech dialogue dataset named \method-500k which includes nearly 500k turns of speech-to-speech dialogues, and a cross-modality training strategy to enhance the semantic alignment between speech and text. Experimental results demonstrate that IntrinsicVoice can generate high-quality speech response with latency lower than 100ms in multi-turn dialogue scenarios. Demos are available at https://instrinsicvoice.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
