arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.07919eess.AScs.CLcs.LG
收藏

Qwen-Audio:通过统一的大规模音频-语言模型推进通用音频理解

Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models

Alibaba Group(阿里巴巴集团)

Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, Jingren Zhou

AI总结:

本文提出Qwen-Audio,通过扩展音频-语言预训练至30多个任务和多种音频类型,并设计分层标签的多任务训练框架解决干扰问题,实现了无需微调的通用音频理解,并进一步开发了Qwen-Audio-Chat支持多轮对话。

摘要:

近年来,遵循指令的音频-语言模型因支持与人类的音频交互而受到广泛关注。然而,缺乏能够处理多种音频类型和任务的预训练音频模型阻碍了该领域的进展。因此,大多数现有工作只能支持有限的交互能力。在本文中,我们开发了Qwen-Audio模型,通过将音频-语言预训练扩展到覆盖超过30个任务和多种音频类型(如人类语音、自然声音、音乐和歌曲)来解决这一限制,以促进通用音频理解能力。然而,直接联合训练所有任务和数据集可能会导致干扰问题,因为不同数据集关联的文本标签因任务重点、语言、注释粒度和文本结构的差异而表现出相当大的变化。为了克服这种一对多的干扰,我们精心设计了一个多任务训练框架,通过将一系列分层标签作为条件输入到解码器,分别利用共享标签和特定标签来鼓励知识共享并避免干扰。值得注意的是,Qwen-Audio在多种基准任务上无需任何任务特定的微调即取得了令人印象深刻的性能,超越了同类模型。基于Qwen-Audio的能力,我们进一步开发了Qwen-Audio-Chat,它允许输入各种音频和文本,支持多轮对话,并适用于多种以音频为中心的场景。

英文摘要:

Recently, instruction-following audio-language models have received broad attention for audio interaction with humans. However, the absence of pre-trained audio models capable of handling diverse audio types and tasks has hindered progress in this field. Consequently, most existing works have only been able to support a limited range of interaction capabilities. In this paper, we develop the Qwen-Audio model and address this limitation by scaling up audio-language pre-training to cover over 30 tasks and various audio types, such as human speech, natural sounds, music, and songs, to facilitate universal audio understanding abilities. However, directly co-training all tasks and datasets can lead to interference issues, as the textual labels associated with different datasets exhibit considerable variations due to differences in task focus, language, granularity of annotation, and text structure. To overcome the one-to-many interference, we carefully design a multi-task training framework by conditioning on a sequence of hierarchical tags to the decoder for encouraging knowledge sharing and avoiding interference through shared and specified tags respectively. Remarkably, Qwen-Audio achieves impressive performance across diverse benchmark tasks without requiring any task-specific fine-tuning, surpassing its counterparts. Building upon the capabilities of Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from various audios and text inputs, enabling multi-turn dialogues and supporting various audio-central scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑