一种基于大语言模型的全双工语音对话方案
A Full-duplex Speech Dialogue Scheme Based On Large Language Models
AI总结：
本研究提出一种基于大语言模型的全双工语音对话系统，通过神经有限状态机实现实时交互决策，在更低延迟的同时实现了更优的打断准确率。
摘要：
我们提出了一种能够以全双工模式运行的生成式对话系统，可实现流畅自然的交互。该系统基于大语言模型（LLM）构建，经过精心对齐，能够感知感知模块、运动功能模块，以及一种被称为neural FSM（神经有限状态机）的双状态简单有限状态机概念。感知模块与运动功能模块协同运作，使系统能够同时向用户讲话并聆听用户发言。LLM会生成用于答复询问的文本token，并且通过向neural FSM发送控制token，自主决定开始响应用户、等待用户或是打断用户。LLM的所有这些任务，都是在对话的序列化视图上以实时下一词预测的方式完成的。在模拟真实交互的自动质量评估中，与基于LLM的半双工对话系统相比，所提出的系统将平均对话响应延迟降低了三倍以上，且在超过50%的评估交互中，响应时间不到500毫秒。我们的系统仅运行参数规模为80亿的LLM，其打断准确率就比现有最优的商用语音对话LLM高出8%。
英文摘要：
We present a generative dialogue system capable of operating in a full-duplex manner, allowing for seamless interaction. It is based on a large language model (LLM) carefully aligned to be aware of a perception module, a motor function module, and the concept of a simple finite state machine (called neural FSM) with two states. The perception and motor function modules operate in tandem, allowing the system to speak and listen to the user simultaneously. The LLM generates textual tokens for inquiry responses and makes autonomous decisions to start responding to, wait for, or interrupt the user by emitting control tokens to the neural FSM. All these tasks of the LLM are carried out as next token prediction on a serialized view of the dialogue in real-time. In automatic quality evaluations simulating real-life interaction, the proposed system reduces the average conversation response latency by more than threefold compared with LLM-based half-duplex dialogue systems while responding within less than 500 milliseconds in more than 50% of evaluated interactions. Running an LLM with only 8 billion parameters, our system exhibits an 8% higher interruption precision rate than the best available commercial LLM for voice-based dialogue.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
