arXiv 2406.100562024-06-14cs.SDeess.AS

UniAudio 1.5：大语言模型驱动的音频编解码器是少样本音频任务学习器

UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learner

The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；Microsoft Research Asia（微软亚洲研究院）

Dongchao Yang, Haohan Guo, Yuanyuan Wang, Rongjie Huang, Xiang Li, Xu Tan, Xixin Wu, Helen Meng

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出LLM-Codec将音频压缩为大语言模型可理解的token，使冻结LLM无需微调即可通过少样本上下文学习完成多种音频理解与生成任务，并开源模型。

摘要：

大语言模型（LLMs）在文本理解与生成方面展现出卓越能力，但若不经过微调，无法直接应用于跨模态任务。本文提出一种跨模态上下文学习方法，使冻结的LLMs能够以少样本方式完成多种音频任务，且无需任何参数更新。具体而言，我们提出一种新颖的、由LLMs驱动的音频编解码模型LLM-Codec，将音频模态转换到文本空间，即用LLMs词汇表中的词或子词来表示音频token，同时保持较高的音频重建质量。其核心思想是通过将音频模态压缩到训练良好的LLMs token空间中，降低文本与音频之间的模态异质性。因此，音频表示可被视为一种新的“外语”，LLMs可以通过若干示例来学习这种新的“外语”。在实验中，我们考察了所提方法在多个音频理解与生成任务上的表现，例如语音情感分类、音频分类、文本到语音生成、语音增强等。实验结果表明，配备所提LLM-Codec的LLMs（命名为UniAudio 1.5）仅通过少量示例提示，即可在简单场景中实现预期功能，验证了所提跨模态上下文学习方法的可行性与有效性。为促进少样本音频任务学习和多模态LLMs研究，我们已开源LLM-Codec模型。

英文摘要：

The Large Language models (LLMs) have demonstrated supreme capabilities in text understanding and generation, but cannot be directly applied to cross-modal tasks without fine-tuning. This paper proposes a cross-modal in-context learning approach, empowering the frozen LLMs to achieve multiple audio tasks in a few-shot style without any parameter update. Specifically, we propose a novel and LLMs-driven audio codec model, LLM-Codec, to transfer the audio modality into the textual space, \textit{i.e.} representing audio tokens with words or sub-words in the vocabulary of LLMs, while keeping high audio reconstruction quality. The key idea is to reduce the modality heterogeneity between text and audio by compressing the audio modality into a well-trained LLMs token space. Thus, the audio representation can be viewed as a new \textit{foreign language}, and LLMs can learn the new \textit{foreign language} with several demonstrations. In experiments, we investigate the performance of the proposed approach across multiple audio understanding and generation tasks, \textit{e.g.} speech emotion classification, audio classification, text-to-speech generation, speech enhancement, etc. The experimental results demonstrate that the LLMs equipped with the proposed LLM-Codec, named as UniAudio 1.5, prompted by only a few examples, can achieve the expected functions in simple scenarios. It validates the feasibility and effectiveness of the proposed cross-modal in-context learning approach. To facilitate research on few-shot audio task learning and multi-modal LLMs, we have open-sourced the LLM-Codec model.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。