arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.05968eess.AScs.CL
收藏

面向通用语音摘要的音频提示大语言模型方法

Prompting Large Language Models with Audio for General-Purpose Speech Summarization

Massachusetts Institute of Technology(麻省理工学院)

Wonjune Kang, Deb Roy

AI总结:

本研究提出结合指令微调LLM与音频编码器的端到端语音摘要框架,经语音-文本配对数据训练后可通过提示实现跨领域多风格语音摘要,性能优于级联基线系统。

摘要:

本研究提出一种利用大语言模型(LLM)的处理与推理能力的语音摘要框架。我们构建了一套端到端系统,将经过指令微调的LLM与音频编码器相结合,音频编码器可将语音转换为LLM能够解读的token表示。系统基于配对的语音-文本数据集训练,目标是针对语义相同的提示,无论输入模态是语音还是文本,都能生成一致的响应。该框架让LLM可以像处理文本一样处理语音输入,仅需对LLM进行提示即可实现语音摘要。与现有方法不同,本方法可对任意领域的语音内容生成摘要,还能通过调整LLM的提示策略生成不同风格的摘要。实验表明,该方法的性能优于“语音识别+LLM文本处理”的级联基线系统。

英文摘要:

In this work, we introduce a framework for speech summarization that leverages the processing and reasoning capabilities of large language models (LLMs). We propose an end-to-end system that combines an instruction-tuned LLM with an audio encoder that converts speech into token representations that the LLM can interpret. Using a dataset with paired speech-text data, the overall system is trained to generate consistent responses to prompts with the same semantic information regardless of the input modality. The resulting framework allows the LLM to process speech inputs in the same way as text, enabling speech summarization by simply prompting the LLM. Unlike prior approaches, our method is able to summarize spoken content from any arbitrary domain, and it can produce summaries in different styles by varying the LLM prompting strategy. Experiments demonstrate that our approach outperforms a cascade baseline of speech recognition followed by LLM text processing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑