在全双工语音模型中释放LLM能力
Liberating LLM Capabilities in Full-Duplex Speech Models
AI总结：
提出Listen-Write-Speak (LWS)三通道范式，使LLM在共享因果注意力上下文中同时监听、书写可见文本并实时口语回应，无需架构修改，实现全双工交互。
摘要：
基于语音的大型语言模型通常局限于口语回复，这将其面向用户的输出限制在可口头表达的内容上，并抑制了文本原生能力，如代码生成、结构化分析和实时交互中的多步推理，对于需要持久、结构化且可检查的中间输出的任务。现有工作改进了口语推理或全双工轮流发言，但仍将文本视为隐藏的中间状态或从属模态，而非第一类输出通道。我们提出Listen-Write-Speak (LWS)，一种文本优先的三通道范式，其中单个自回归LLM持续监听用户音频，写出可见的自由形式文本作为其主要输出，并在共享因果注意力上下文中并行生成实时口语回应。该行为完全通过Token Schema实现，无需架构修改，并通过两阶段数据流水线学习，该流水线合成与揭示的输入时间线一致的每秒认知注释。实验上，LWS在Full-Duplex-Bench上展示了强大的全双工交互，在VoiceBench AlpacaEval上达到4.72，写作-口语一致性达92.6%，并在URO-Bench上持续优于其内部消融版本。这些结果表明，可见书写可以作为语音交互的第一类输出通道，而不会牺牲实时响应性。代码和数据集可在项目页面获取：https://royalzhang.com/project/lws-page/。
英文摘要：
Speech-based large language models are typically constrained to spoken replies, which limits their user-facing outputs to what can be verbalized and suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning in realtime interaction, for tasks that require persistent, structured, and inspectable intermediate outputs. Existing work improves spoken reasoning or full-duplex turn-taking, but still treats text as a hidden intermediate state or a subordinate modality rather than a first-class output channel. We propose Listen-Write-Speak (LWS), a text-first tri-channel paradigm in which a single autoregressive LLM continuously listens to user audio, writes visible free-form text as its primary output, and speaks a realtime oral response in parallel under a shared causal attention context. This behavior is implemented entirely through a Token Schema, requiring no architectural modifications, and learned via a two-stage data pipeline that synthesizes per-second cognitive annotations consistent with the revealed input timeline. Empirically, LWS demonstrates strong full-duplex interaction on Full-Duplex-Bench, reaches 4.72 on VoiceBench AlpacaEval, achieves 92.6% writing-speaking consistency, and consistently outperforms its internal ablations on URO-Bench. These results suggest that visible writing can serve as a first-class output channel for speech interaction without sacrificing realtime responsiveness. The code and dataset are available on the project page: https://royalzhang.com/project/lws-page/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
