arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.08846cs.CLcs.AIcs.MMcs.SDeess.AS
收藏

一种为 LLM 赋予强 ASR 能力的极其简单的方法

An Embarrassingly Simple Approach for LLM with Strong ASR Capacity

Alibaba Group(阿里巴巴集团)

Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, Xie Chen

AI总结:

本文提出SLAM-ASR,仅训练线性投影器来连接现成的语音编码器与LLM,在Librispeech基准上取得了基于LLM的ASR模型最佳性能。

摘要:

本文聚焦于使用语音基础编码器和大型语言模型(LLM)来解决语音处理领域最重要的任务之一,即自动语音识别(ASR)。近期的研究采用了复杂的设计,例如对语音编码器的输出进行时间压缩、为投影器解决模态对齐问题,以及为 LLM 利用参数高效微调。我们发现精巧的设计并非必要,仅需将现成的语音编码器、LLM 以及唯一可训练的线性投影器进行极其简单的组合,便能胜任 ASR 任务。具体而言,我们对各种 LLM 与语音编码器的组合进行了基准测试和探索,从而得出了最优的基于 LLM 的 ASR 系统,我们称之为 SLAM-ASR。提出的 SLAM-ASR 提供了干净的设置且几乎没有任务特定的设计,其中仅训练线性投影器。据我们所知,SLAM-ASR 在基于 LLM 的 ASR 模型中于 Librispeech 基准上取得了最佳性能,甚至超越了在海量配对数据上训练的最新基于 LLM 的音频通用模型。最后,我们探索了基于 LLM 的 ASR 在模态对齐过程中的能力涌现。我们希望我们的研究能促进扩展 LLM 跨模态能力的研究,并为基于 LLM 的 ASR 社区提供启示。

英文摘要:

In this paper, we focus on solving one of the most important tasks in the field of speech processing, i.e., automatic speech recognition (ASR), with speech foundation encoders and large language models (LLM). Recent works have complex designs such as compressing the output temporally for the speech encoder, tackling modal alignment for the projector, and utilizing parameter-efficient fine-tuning for the LLM. We found that delicate designs are not necessary, while an embarrassingly simple composition of off-the-shelf speech encoder, LLM, and the only trainable linear projector is competent for the ASR task. To be more specific, we benchmark and explore various combinations of LLMs and speech encoders, leading to the optimal LLM-based ASR system, which we call SLAM-ASR. The proposed SLAM-ASR provides a clean setup and little task-specific design, where only the linear projector is trained. To the best of our knowledge, SLAM-ASR achieves the best performance on the Librispeech benchmark among LLM-based ASR models and even outperforms the latest LLM-based audio-universal model trained on massive pair data. Finally, we explore the capability emergence of LLM-based ASR in the process of modal alignment. We hope that our study can facilitate the research on extending LLM with cross-modality capacity and shed light on the LLM-based ASR community.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑