Recently, instruction-following audio-language models have received broad attention for audio interaction with humans. However, the absence of pre-trained audio models capable of handling diverse audio types and tasks has hindered progress in this field. Consequently, most existing works have only been able to support a limited range of interaction capabilities. In this paper, we develop the Qwen-Audio model and address this limitation by scaling up audio-language pre-training to cover over 30 tasks and various audio types, such as human speech, natural sounds, music, and songs, to facilitate universal audio understanding abilities. However, directly co-training all tasks and datasets can lead to interference issues, as the textual labels associated with different datasets exhibit considerable variations due to differences in task focus, language, granularity of annotation, and text structure. To overcome the one-to-many interference, we carefully design a multi-task training framework by conditioning on a sequence of hierarchical tags to the decoder for encouraging knowledge sharing and avoiding interference through shared and specified tags respectively. Remarkably, Qwen-Audio achieves impressive performance across diverse benchmark tasks without requiring any task-specific fine-tuning, surpassing its counterparts. Building upon the capabilities of Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from various audios and text inputs, enabling multi-turn dialogues and supporting various audio-central scenarios.
CMDAR: A Chinese Multi-scene Dynamic Audio Reasoning Benchmark with Diverse Challenges
College of Computer Science and Artificial Intelligence, Fudan University(计算机科学与人工智能学院,复旦大学);Shanghai Jiao Tong University(上海交通大学);IEIT Systems Co Ltd(IEIT系统有限公司)
HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
EPIC Lab, Shanghai Jiao Tong University(上海交通大学EPIC实验室);DAIL Tech(DAIL科技);Northeastern University(东北大学);Sichuan University(四川大学);Huazhong University of Science and Technology(华中科技大学)
AHA: Aligning Large Audio-Language Models for Reasoning Hallucinations via Counterfactual Hard Negatives
Arizona State University(亚利桑那州立大学);Clemson University(克莱姆斯大学);Washington University in St.Louis(圣路易斯华盛顿大学);Rice University(Rice 大学);Morgan Stanley(摩根大通)
Distinct social-linguistic processing between humans and large audio-language models: Evidence from model-brain alignment
Department of Linguistics and Modern Languages, The Chinese University of Hong Kong(语言学与现代语言系,香港中文大学);Brain and Mind Institute, The Chinese University of Hong Kong(脑与心智研究所,香港中文大学)
Words at Play: Benchmarking Audio Pun Understanding in Large Audio-Language Models
School of Computer Science, University of Auckland(奥克兰大学计算机科学学院);Singapore University of Technology and Design(新加坡科技设计大学);School of Electronic Engineering and Computer Science, Queen Mary University of London(伦敦大学玛丽女王学院电子工程与计算机科学学院);School of Statistics and Data Science, Nankai University(南开大学统计与数据科学学院)
Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation
Columbia University(哥伦比亚大学);University of Washington(华盛顿大学)
Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models
National University of Singapore(新加坡国立大学);ASUS Intelligent Cloud Services(华硕智能云服务);I2R, Agency for Science, Technology, and Research (A*STAR)(新加坡科技研究局信息研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Universidad Politécnica de Madrid(马德里理工大学)
Investigating Faithfulness in Large Audio Language Models
Concordia University(康科迪亚大学);Mila - Quebec AI Institute(魁北克人工智能研究院);Université Laval(拉瓦尔大学);Birla Institute of Technology and Science, Pilani(比拉理工学院和科学学院,皮兰尼)
Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination
Sharif University of Technology(谢里夫理工大学);Amirkabir University of Technology(阿米尔卡比尔理工大学);New Uzbekistan University(新乌兹别克斯坦大学);Okinawa Institute of Science and Technology(冲绳科学技术大学院大学)