通过激活引导克服全双工口语语言模型中的状态惯性
Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering
AI总结：
针对全双工口语模型在用户打断时响应延迟的问题，提出基于感知向量的激活引导方法，无需微调即可显著提升中断理解能力。
摘要：
全双工口语语言模型（FD-SLMs）通过允许模型同时听和说实现无缝语音交互，但其协调听与说的内部机制尚未充分探索。我们分析了FD-SLM隐藏表示中编码的预测行为，发现它们表现出特定流的预测模式：在听时，它们优先预测传入的用户流；而在说时，它们优先预测模型输出流。基于这一观察，我们表明FD-SLMs动态调节其内部预测焦点在两个状态之间：与模型输出生成一致的生成状态和与传入用户输入一致的感知状态。然而，这种调节可能滞后于对话上下文的突然变化。在用户打断期间，模型在过渡到感知状态之前短暂地偏向生成状态，导致其错过传入输入的开头。我们将这种延迟的内部过渡称为状态惯性。为了量化其下游影响，我们引入了零缓冲基准（ZBB），这是一个用于评估当用户语音突然开始时即时中断理解能力的诊断基准。我们使用响应正确性和初始词出现率（IWOR）来评估这一设置。最后，我们通过使用感知向量的激活引导来缓解状态惯性，这是一种无需训练且计算开销很小的干预措施。在多个最先进的FD-SLMs上，激活引导显著改善了中断处理；例如，在PersonaPlex上，它将正确性从28%提高到45%，将IWOR从40%提高到72%，而无需任何微调。
英文摘要：
Full-duplex spoken language models (FD-SLMs) enable seamless speech interaction by allowing models to listen and speak simultaneously, yet the internal mechanism by which they coordinate listening and speaking remains underexplored. We analyze the predictive behavior encoded in FD-SLM hidden representations and find that they exhibit stream-specific predictive patterns: during listening, they preferentially predict the incoming user stream, whereas during speaking, they preferentially predict the model output stream. Building on this observation, we show that FD-SLMs dynamically modulate their internal predictive focus between two states: a generative state aligned with model output generation and a perceptive state aligned with incoming user input. However, this modulation can lag behind abrupt changes in conversational context. During user interruptions, the model remains transiently biased toward the generative state before transitioning into the perceptive state, causing it to miss the beginning of the incoming input. We term this delayed internal transition state inertia. To quantify its downstream impact, we introduce the Zero-Buffer Benchmark (ZBB), a diagnostic benchmark for evaluating immediate interruption comprehension when user speech begins abruptly. We evaluate this setting using response correctness and initial-word occurrence rate (IWOR). Finally, we mitigate state inertia through activation steering with a perception vector, a training-free intervention with little additional computational overhead. Across multiple state-of-the-art FD-SLMs, activation steering substantially improves interruption handling; for example, on PersonaPlex, it improves correctness from 28% to 45% and IWOR from 40% to 72% without any fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
