SALM：具有上下文学习能力的语音增强语言模型，用于语音识别和翻译
SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation
AI总结：
提出SALM模型，结合冻结文本LLM、音频编码器和LoRA层，实现多任务和上下文学习，在ASR和AST上性能与Conformer相当，并通过语音监督上下文训练增强零样本学习能力，已开源。
摘要：
我们提出了一种新颖的语音增强语言模型（SALM），具有多任务和上下文学习能力。SALM由一个冻结的文本大语言模型（LLM）、一个音频编码器、一个模态适配器模块以及LoRA层组成，以适应语音输入和相关任务指令。统一的SALM不仅在自动语音识别（ASR）和语音翻译（AST）方面达到了与特定任务Conformer基线相当的性能，而且还展示了零样本上下文学习能力，通过ASR和AST的关键词增强任务得到了证明。此外，我们提出了语音监督的上下文训练，以弥合LLM训练与下游语音任务之间的差距，这进一步增强了语音到文本模型的上下文学习能力。所提出的模型通过NeMo工具包开源。
英文摘要：
We present a novel Speech Augmented Language Model (SALM) with {\em multitask} and {\em in-context} learning capabilities. SALM comprises a frozen text LLM, a audio encoder, a modality adapter module, and LoRA layers to accommodate speech input and associated task instructions. The unified SALM not only achieves performance on par with task-specific Conformer baselines for Automatic Speech Recognition (ASR) and Speech Translation (AST), but also exhibits zero-shot in-context learning capabilities, demonstrated through keyword-boosting task for ASR and AST. Moreover, {\em speech supervised in-context training} is proposed to bridge the gap between LLM training and downstream speech tasks, which further boosts the in-context learning ability of speech-to-text models. Proposed model is open-sourced via NeMo toolkit.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
