SHANKS：面向口语语言模型的同步聆听与思考
SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models
AI总结：
本文提出 SHANKS 推理框架，让口语语言模型在用户说话时流式生成隐含思维链，并据此实时打断或预取工具调用，显著提升交互实时性。
摘要：
当前的大语言模型（LLM）和口语语言模型（SLM）只有在用户完成其话轮之后才开始思考并采取行动。这使模型无法在用户说话期间进行交互，并且由于其等待后才思考，可能导致较高的响应延迟。因此，在接收到完整输入后再思考并不适合重视实时、低延迟交流的语音到语音交互。我们通过注意到人类会自然地“边听边思考”来解决这一问题。本文提出 SHANKS，这是一个通用推理框架，使 SLM 能够在聆听用户输入的同时生成未说出的思维链推理。SHANKS 将输入语音按固定时长的分块进行流式处理；一旦接收到一个分块，就在用户继续说话的同时，基于此前所有语音和推理生成未说出的推理。SHANKS 使用这种未说出的推理来决定是否打断用户，以及进行工具调用以完成任务。我们证明 SHANKS 在两种场景中增强了实时的用户–SLM 交互：（1）当用户逐步讲解一道数学题的解法时，SHANKS 能够聆听、推理，并在用户出错时打断，相比不经过思考就打断的基线，其打断准确率高出 37.1%；（2）在工具增强对话中，SHANKS 能够在用户结束话轮之前完成 56.9% 的工具调用。总体而言，SHANKS 推动模型走向在整个对话过程中持续思考，而不仅是在话轮结束后才思考。SHANKS 的动画演示可在 https://d223302.github.io/SHANKS/ 查看。
英文摘要：
Current large language models (LLMs) and spoken language models (SLMs) begin thinking and taking actions only after the user has finished their turn. This prevents the model from interacting during the user's turn and can lead to high response latency while it waits to think. Consequently, thinking after receiving the full input is not suitable for speech-to-speech interaction, where real-time, low-latency exchange is important. We address this by noting that humans naturally "think while listening." In this paper, we propose SHANKS, a general inference framework that enables SLMs to generate unspoken chain-of-thought reasoning while listening to the user input. SHANKS streams the input speech in fixed-duration chunks and, as soon as a chunk is received, generates unspoken reasoning based on all previous speech and reasoning, while the user continues speaking. SHANKS uses this unspoken reasoning to decide whether to interrupt the user and to make tool calls to complete the task. We demonstrate that SHANKS enhances real-time user-SLM interaction in two scenarios: (1) when the user is presenting a step-by-step solution to a math problem, SHANKS can listen, reason, and interrupt when the user makes a mistake, achieving 37.1% higher interruption accuracy than a baseline that interrupts without thinking; and (2) in a tool-augmented dialogue, SHANKS can complete 56.9% of the tool calls before the user finishes their turn. Overall, SHANKS moves toward models that keep thinking throughout the conversation, not only after a turn ends. Animated illustrations of Shanks can be found at https://d223302.github.io/SHANKS/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
