为LLM配备方向性多说话人语音理解能力
Equipping LLM with Directional Multi-Talker Speech Understanding Capabilities
AI总结：
本文提出两种方法，通过多麦克风阵列和级联系统，使LLM具备方向性多说话人语音理解能力，提升了语音识别和翻译性能。
摘要：
近期研究表明，通过提示大型语言模型（LLM）使用音频编码可以实现有效的语音理解能力。然而，大多数语音LLM都是在单通道、单说话人数据上训练的，这使得它们难以直接应用于多说话人和多通道语音理解任务。在本工作中，我们全面探讨了如何为LLM启用方向性多说话人语音理解能力，特别是在智能眼镜的使用场景中。我们提出了两种新颖的方法来将方向性集成到LLM中：（1）一个级联系统，利用源分离前端模块；（2）一个端到端系统，利用序列化输出训练。所有方法都利用嵌入在智能眼镜中的多麦克风阵列，在流式处理中优化方向性解释和处理。实验结果证明了我们提出的方法在赋予LLM方向性语音理解能力方面的有效性，在语音识别和语音翻译任务中均取得了强劲的性能。
英文摘要：
Recent studies have demonstrated that prompting large language models (LLM) with audio encodings enables effective speech understanding capabilities. However, most speech LLMs are trained on single-channel, single-talker data, which makes it challenging to directly apply them to multi-talker and multi-channel speech understanding task. In this work, we present a comprehensive investigation on how to enable directional multi-talker speech understanding capabilities for LLMs, specifically in smart glasses usecase. We propose two novel approaches to integrate directivity into LLMs: (1) a cascaded system that leverages a source separation front-end module, and (2) an end-to-end system that utilizes serialized output training. All of the approaches utilize a multi-microphone array embedded in smart glasses to optimize directivity interpretation and processing in a streaming manner. Experimental results demonstrate the efficacy of our proposed methods in endowing LLMs with directional speech understanding capabilities, achieving strong performance in both speech recognition and speech translation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
