In the era of large language models (LLMs) and artificial general intelligence (AGI), computer audition must evolve beyond traditional paradigms to fully leverage the capabilities of foundation models, towards more comprehensive understanding, more natural generation and more human-like interaction. Audio, as a modality rich in semantic, emotional, and contextual cues, plays a vital role in achieving naturalistic and embodied machine intelligence. This survey provides a comprehensive review of recent progress in integrating audio into LLMs, with a focus on four key areas: audio comprehension, audio generation, speech-based interaction, and audio-visual understanding. We analyze how LLMs are reshaping audio perception and reasoning, enabling systems to understand sound at a deeper semantic level, generate expressive audio outputs, and engage in human-like spoken interaction. Furthermore, we explore how the fusion of audio and visual modalities enhances situational awareness and cross-modal reasoning, pushing the boundaries of multimodal intelligence. This survey not only synthesizes existing research but also identifies critical challenges and future directions for building audio-native AGI systems capable of perceiving, understanding, and interacting through sound as naturally as humans do.
DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations
Renmin University of China(中国人民大学);Ant Group(蚂蚁集团);Tsinghua University(清华大学);Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing(北京未来区块链与隐私计算创新中心)
Development of the Listening in Spatialized Noise-Sentences (LiSN-S) Test in Brazilian Portuguese: Presentation Software, Speech Stimuli, and Sentence Equivalence
University of Campinas(坎皮纳斯大学);Pontifical Catholic University of Campinas(坎皮纳斯天主教大学);Macquarie University(麦考瑞大学)
Listening, Imagining & Refining: A Heuristic Optimized ASR Correction Framework with LLMs
School of Information Science and Technology, Tibet University(信息科学与技术学院,西藏大学);Department of Ophthalmology, University of Texas Southwestern Medical Center(眼科部门,德克萨斯西南医学中心)
AAD-LLM: Neural Attention-Driven Auditory Scene Understanding
Department of Electrical Engineering(电气工程系);Department of Neurological Surgery(神经外科系);Mortimer B. Zuckerman Mind Brain Behavior Institute(莫提默·B·祖克曼心智-脑-行为研究所);Hofstra Northwell School of Medicine(霍夫曼斯特拉北well医学院);The Feinstein Institutes for Medical Research(费森特医学研究所);Neurology Department(神经病学系)
Learn to Sing by Listening: Building Controllable Virtual Singer by Unsupervised Learning from Voice Recordings
Hong Kong University of Science and Technology(香港科技大学);Hong Kong Chu Hai College(香港珠海学院);Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院)
Unsupervised Auditory and Semantic Entrainment Models with Deep Neural Networks
Institute of Informatics, Slovak Academy of Sciences(斯洛伐克科学院信息学研究所);Faculty of Informatics and Information Technology, Slovak Technical University(斯洛伐克技术大学信息学与信息技术学院);Constantine the Philosopher University, Nitra(尼特拉君士坦丁哲学家大学);Orange Innovation(Orange创新)
Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realism
National Technical University of Athens(雅典国家技术大学);Athena Research Center(雅典娜研究中心);Foundation for Research & Technology - Hellas (FORTH)(希腊研究与技术基金会(FORTH))