arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.01299eess.AS
收藏

迈向通用听觉智能:用于机器聆听与说话的大型多模态模型

Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking

Tsinghua University(清华大学);Google DeepMind(谷歌DeepMind);NVIDIA(英伟达);University of Cambridge(剑桥大学)
Siyin Wang, Zengrui Jin, Changli Tang, Qiujia Li, Bo Li, Chen Chen, Yuchen Hu, Wenyi Yu, Yixuan Li, Jimin Zhuang, Yudong Yang, Mingqiu Wang, Michael Han, Yifan … 展开作者

Siyin Wang, Zengrui Jin, Changli Tang, Qiujia Li, Bo Li, Chen Chen, Yuchen Hu, Wenyi Yu, Yixuan Li, Jimin Zhuang, Yudong Yang, Mingqiu Wang, Michael Han, Yifan Ding, Junwen Bai, Tom Ouyang, Shuo-yiin Chang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Guangzhi Sun, Zhehuai Chen, Ji Wu, Bowen Zhou, Yuxuan Wang, Tara Sainath, Yonghui Wu, Chao Zhang

AI总结:

本文综述了将音频整合到大型语言模型中的最新进展,重点探讨音频理解、生成、语音交互及视听理解,指出了构建原生音频AGI系统的挑战与未来方向。

摘要:

在大型语言模型(LLMs)和通用人工智能(AGI)时代,计算机听觉必须超越传统范式,以充分利用基础模型的能力,实现更全面的理解、更自然的生成和更类人的交互。音频作为一种富含语义、情感和上下文线索的模态,在实现自然和具身的机器智能中发挥着至关重要的作用。本综述全面回顾了将音频整合到LLMs中的最新进展,重点关注四个关键领域:音频理解、音频生成、基于语音的交互以及视听理解。我们分析了LLMs如何重塑音频感知与推理,使系统能够在更深层的语义级别上理解声音,生成富有表现力的音频输出,并参与类人的语音交互。此外,我们探讨了音频与视觉模态的融合如何增强情境感知和跨模态推理,从而拓展多模态智能的边界。本综述不仅综合了现有研究,还指出了构建原生音频AGI系统的关键挑战和未来方向,该系统能够像人类一样自然地通过声音进行感知、理解和交互。

英文摘要:

In the era of large language models (LLMs) and artificial general intelligence (AGI), computer audition must evolve beyond traditional paradigms to fully leverage the capabilities of foundation models, towards more comprehensive understanding, more natural generation and more human-like interaction. Audio, as a modality rich in semantic, emotional, and contextual cues, plays a vital role in achieving naturalistic and embodied machine intelligence. This survey provides a comprehensive review of recent progress in integrating audio into LLMs, with a focus on four key areas: audio comprehension, audio generation, speech-based interaction, and audio-visual understanding. We analyze how LLMs are reshaping audio perception and reasoning, enabling systems to understand sound at a deeper semantic level, generate expressive audio outputs, and engage in human-like spoken interaction. Furthermore, we explore how the fusion of audio and visual modalities enhances situational awareness and cross-modal reasoning, pushing the boundaries of multimodal intelligence. This survey not only synthesizes existing research but also identifies critical challenges and future directions for building audio-native AGI systems capable of perceiving, understanding, and interacting through sound as naturally as humans do.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑