SALMONN：迈向大语言模型的通用听觉能力
SALMONN: Towards Generic Hearing Abilities for Large Language Models
AI总结：
本文提出SALMONN多模态模型，将文本LLM与语音及音频编码器融合，使其具备通用听觉理解、多任务竞争性能和跨模态涌现能力，并用少样本激活调优激发这些能力。
摘要：
听觉可以说是物理世界中人工智能（AI）智能体的一项基本能力，它指的是对至少包含三类声音的一般听觉信息进行感知和理解：语音、音频事件和音乐。本文提出了SALMONN，即语音音频语言音乐开放神经网络，它通过将预训练的基于文本的大语言模型（LLM）与语音和音频编码器集成到单一多模态模型中而构建。SALMONN使LLM能够直接处理和理解通用音频输入，并在训练中使用的多项语音和音频任务上取得具有竞争力的表现，例如自动语音识别和翻译、基于听觉信息的问答、情感识别、说话人验证以及音乐和音频字幕生成等。SALMONN还具备训练中未曾见过的多种涌现能力，包括但不限于向未训练语言进行语音翻译、基于语音的槽位填充、基于口语查询的问答、基于音频的讲故事以及语音音频协同推理等。本文研究了跨模态涌现能力的存在，并提出了一种新颖的少样本激活调优方法来激活此类能力。据我们所知，SALMONN是同类型中的首个模型，可被视为迈向具有通用听觉能力的AI的一步。源代码、模型检查点和数据可在https://github.com/bytedance/SALMONN获取。
英文摘要：
Hearing is arguably an essential ability of artificial intelligence (AI) agents in the physical world, which refers to the perception and understanding of general auditory information consisting of at least three types of sounds: speech, audio events, and music. In this paper, we propose SALMONN, a speech audio language music open neural network, built by integrating a pre-trained text-based large language model (LLM) with speech and audio encoders into a single multimodal model. SALMONN enables the LLM to directly process and understand general audio inputs and achieve competitive performances on a number of speech and audio tasks used in training, such as automatic speech recognition and translation, auditory-information-based question answering, emotion recognition, speaker verification, and music and audio captioning etc. SALMONN also has a diverse set of emergent abilities unseen in the training, which includes but is not limited to speech translation to untrained languages, speech-based slot filling, spoken-query-based question answering, audio-based storytelling, and speech audio co-reasoning etc. The presence of cross-modal emergent abilities is studied, and a novel few-shot activation tuning approach is proposed to activate such abilities. To our knowledge, SALMONN is the first model of its type and can be regarded as a step towards AI with generic hearing abilities. The source code, model checkpoints and data are available at https://github.com/bytedance/SALMONN.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
