大语言模型能够根据多样化指令在多说话人场景中转录语音
Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions
AI总结：
本研究提出MT-LLM系统，利用WavLM和Whisper编码器提取语音表示，并通过LoRA微调LLM，使其能根据多样化指令在多说话人环境中进行语音转录，在鸡尾酒会场景中表现优异。
摘要：
大型语言模型（LLM）的最新进展彻底改变了各个领域，带来了显著的进步和新机遇。尽管在语音相关任务方面取得了进展，但 LLM 在多说话人场景中尚未得到充分探索。在这项工作中，我们提出了一项开创性研究，探讨 LLM 在多说话人环境中转录语音的能力，遵循与多说话人自动语音识别（ASR）、目标说话人 ASR，以及基于特定说话人属性（如性别、出现顺序、语言和所说关键词）的 ASR 相关的多样化指令。我们的方法利用 WavLM 和 Whisper 编码器提取对说话人特征和语义上下文敏感的多方面语音表示。然后将这些表示输入到使用 LoRA 微调的 LLM 中，从而实现语音理解和转录的能力。综合实验揭示了提出的系统 MT-LLM 在鸡尾酒会场景中的良好性能，突显了 LLM 在此类复杂环境中基于用户指令处理语音相关任务的潜力。代码、模型和样本可在 https://github.com/cuhealthybrains/MT-LLM 获取。
英文摘要：
Recent advancements in large language models (LLMs) have revolutionized various domains, bringing significant progress and new opportunities. Despite progress in speech-related tasks, LLMs have not been sufficiently explored in multi-talker scenarios. In this work, we present a pioneering effort to investigate the capability of LLMs in transcribing speech in multi-talker environments, following versatile instructions related to multi-talker automatic speech recognition (ASR), target talker ASR, and ASR based on specific talker attributes such as sex, occurrence order, language, and keyword spoken. Our approach utilizes WavLM and Whisper encoder to extract multi-faceted speech representations that are sensitive to speaker characteristics and semantic context. These representations are then fed into an LLM fine-tuned using LoRA, enabling the capabilities for speech comprehension and transcription. Comprehensive experiments reveal the promising performance of our proposed system, MT-LLM, in cocktail party scenarios, highlighting the potential of LLM to handle speech-related tasks based on user instructions in such complex settings. The code, model, and samples are available at https://github.com/cuhealthybrains/MT-LLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
