超越基于回合的界面：作为全双工对话代理的同步LLM
Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents
AI总结：
提出Synchronous LLMs，通过将时间信息整合到Llama3-8b并利用212k小时合成与2k小时真实语音数据训练，实现了超越现有最先进方法的全双工语音对话代理。
摘要：
尽管人们对建模语音对话代理有广泛兴趣，但大多数方法本质上是“半双工”的——受限于基于回合的交互，其响应需要用户显式提示，或对打断与静音事件进行隐式追踪。相比之下，人类对话是“全双工”的，允许以快速动态的轮流发言、重叠语音和后向信道的形式实现丰富的同步性。在技术上，使用LLM实现全双工对话的挑战在于对同步性的建模，因为预训练的LLM没有“时间”概念。为了弥合这一差距，我们提出了用于全双工语音对话建模的Synchronous LLMs。我们设计了一种新颖的机制，将时间信息整合到Llama3-8b中，使其能与现实世界时钟同步运行。我们还引入了一种训练方案，使用从文本对话数据生成的212k小时合成语音对话数据，创建了一个能生成有意义且自然语音对话的模型，而仅需2k小时的真实世界语音对话数据。Synchronous LLMs在对话意义性上超越了现有最先进方法，同时保持了自然度。最后，我们通过模拟在两个不同数据集上训练的代理之间的交互，并考虑高达240 ms的互联网级延迟，展示了该模型参与全双工对话的能力。
英文摘要：
Despite broad interest in modeling spoken dialogue agents, most approaches are inherently "half-duplex" -- restricted to turn-based interaction with responses requiring explicit prompting by the user or implicit tracking of interruption or silence events. Human dialogue, by contrast, is "full-duplex" allowing for rich synchronicity in the form of quick and dynamic turn-taking, overlapping speech, and backchanneling. Technically, the challenge of achieving full-duplex dialogue with LLMs lies in modeling synchrony as pre-trained LLMs do not have a sense of "time". To bridge this gap, we propose Synchronous LLMs for full-duplex spoken dialogue modeling. We design a novel mechanism to integrate time information into Llama3-8b so that they run synchronously with the real-world clock. We also introduce a training recipe that uses 212k hours of synthetic spoken dialogue data generated from text dialogue data to create a model that generates meaningful and natural spoken dialogue, with just 2k hours of real-world spoken dialogue data. Synchronous LLMs outperform state-of-the-art in dialogue meaningfulness while maintaining naturalness. Finally, we demonstrate the model's ability to participate in full-duplex dialogue by simulating interaction between two agents trained on different datasets, while considering Internet-scale latencies of up to 240 ms. Webpage: https://syncllm.cs.washington.edu/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
