语音大语言模型能否边听边思考？
Can Speech LLMs Think while Listening?
AI总结：
针对语音大语言模型推理能力不足与响应延迟高的问题，提出基于文本空间思维链微调提升准确率2.4倍，并利用熵度量“问题完整性”及DPO优化，实现延迟降低70%且不损准确率。
摘要：
近期语音大语言模型（speech LLMs）的进展实现了无缝的语音交互，但这些系统在处理复杂推理任务时仍存在困难。此前，思维链（chain-of-thought, CoT）提示或微调已被证明能显著提升基于文本的大语言模型的推理能力。在本工作中，我们研究了CoT微调对多流语音大语言模型的影响，证明在文本空间中推理能将语音大语言模型在一组口语推理任务上的平均准确率提升2.4倍。除准确率外，语音响应的延迟也是与基于语音的智能体交互的关键因素。受人类“边听边思考”行为的启发，我们提出了减少推理带来的额外延迟的方法，允许模型在用户查询结束前就开始推理。为实现这一点，我们引入了一种基于熵的度量——“问题完整性”，作为指导模型在最佳时间开始推理的指标。与基于启发式的方法相比，该方法能更好地控制准确率-延迟权衡，并且在等效延迟条件下，在ARC-Easy上获得了4%的准确率提升。最后，我们使用通过拒绝采样创建的偏好数据应用直接偏好优化（DPO），进一步推动准确率-延迟帕累托前沿，在不损失准确率的情况下将延迟降低了70%。
英文摘要：
Recent advances in speech large language models (speech LLMs) have enabled seamless spoken interactions, but these systems still struggle with complex reasoning tasks. Previously, chain-of-thought (CoT) prompting or fine-tuning has been to shown to significantly improve the reasoning abilities of text-based LLMs. In this work, we investigate the effect of CoT fine-tuning for multi-stream speech LLMs, demonstrating that reasoning in text space improves the accuracy of speech LLMs by 2.4x, on average, over a suite of spoken reasoning tasks. Beyond accuracy, the latency of the spoken response is a crucial factor for interacting with voice-based agents. Inspired by the human behavior of "thinking while listening," we propose methods to reduce the additional latency from reasoning by allowing the model to start reasoning before the user query has ended. To achieve this, we introduce an entropy-based metric, "question completeness," which acts as an indicator to guide the model on the optimal time to start reasoning. This method provides greater control over the accuracy-latency trade-off compared with heuristic-based approaches and, under equivalent latency conditions, yields a 4% accuracy gain on ARC-Easy. Finally, we use Direct Preference Optimization (DPO) on preference data created using rejection sampling to push the accuracy-latency pareto frontier further, resulting in a 70% reduction in latency without loss in accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
