OmniFlatten：面向无缝语音对话的端到端GPT模型
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
AI总结：
本文提出端到端GPT模型OmniFlatten，通过多阶段后训练和扁平化数据操作将文本LLM适配为全双工语音对话模型，实现低延迟自然双向对话。
摘要：
全双工口语对话系统显著优于传统的轮次式对话系统，因为它们允许同时进行双向通信，更接近人与人之间的交互。然而，在全双工对话系统中实现低延迟和自然交互仍然是一个重大挑战，尤其是考虑到人类对话中的打断、反向通道和重叠语音等动态特性。在本文中，我们提出了一种新颖的基于端到端GPT的模型OmniFlatten，用于全双工对话，能够以低延迟有效建模自然对话中固有的复杂行为。为了实现全双工对话能力，我们提出了一种多阶段后训练方案，逐步将文本大语言模型（LLM）骨干适配为语音-文本对话LLM，能够实时生成文本和语音，而无需修改骨干LLM的架构。训练过程包括三个阶段：模态对齐、半双工对话学习和全双工对话学习。在所有训练阶段中，我们使用扁平化操作对数据进行标准化，这使得在不同模态和任务之间统一训练方法和GPT骨干成为可能。我们的方法为开发高效且自然的端到端全双工口语对话系统提供了一种简单的建模技术和一个有前景的研究方向。OmniFlatten生成的对话音频样本可在该网站（https://omniflatten.github.io/）找到。
英文摘要：
Full-duplex spoken dialogue systems significantly surpass traditional turn-based dialogue systems, as they allow simultaneous bidirectional communication, closely mirroring human-human interactions. However, achieving low latency and natural interactions in full-duplex dialogue systems remains a significant challenge, especially considering human conversation dynamics such as interruptions, backchannels, and overlapping speech. In this paper, we introduce a novel End-to-End GPT-based model OmniFlatten for full-duplex conversation, capable of effectively modeling the complex behaviors inherent to natural conversations with low latency. To achieve full-duplex conversation capabilities, we propose a multi-stage post-training scheme that progressively adapts a text large language model (LLM) backbone into a speech-text dialogue LLM, capable of generating text and speech in real time, without modifying the architecture of the backbone LLM. The training process comprises three stages: modality alignment, half-duplex dialogue learning, and full-duplex dialogue learning. In all training stages, we standardize the data using a flattening operation, which enables unifying the training methods and the GPT backbone across different modalities and tasks. Our approach offers a simple modeling technique and a promising research direction for developing efficient and natural end-to-end full-duplex spoken dialogue systems. Audio samples of dialogues generated by OmniFlatten can be found at this web site (https://omniflatten.github.io/).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
