Although deep reinforcement learning (DRL) approaches in audio signal processing have seen substantial progress in recent years, audio-driven DRL for tasks such as navigation, gaze control and head-orientation control in the context of human-robot interaction have received little attention. Here, we propose an audio-driven DRL framework in which we utilise deep Q-learning to develop an autonomous agent that orients towards a talker in the acoustic environment based on stereo speech recordings. Our results show that the agent learned to perform the task at a near perfect level when trained on speech segments in anechoic environments (that is, without reverberation). The presence of reverberation in naturalistic acoustic environments affected the agent's performance, although the agent still substantially outperformed a baseline, randomly acting agent. Finally, we quantified the degree of generalization of the proposed DRL approach across naturalistic acoustic environments. Our experiments revealed that policies learned by agents trained on medium or high reverb environments generalized to low reverb environments, but policies learned by agents trained on anechoic or low reverb environments did not generalize to medium or high reverb environments. Taken together, this study demonstrates the potential of audio-driven DRL for tasks such as head-orientation control and highlights the need for training strategies that enable robust generalization across environments for real-world audio-driven DRL applications.
Do Models Hear Like Us? Probing the Representational Alignment of Audio LLMs and Naturalistic EEG
School of Computer Science, Chongqing University(重庆大学计算机科学学院);Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua University(清华大学计算机科学与技术系、人工智能研究院);School of Economics and Business Administration, Chongqing University(重庆大学经济与商业管理学院);School of Artificial Intelligence, Southwest University(西南大学人工智能学院);The First Affiliated Hospital of Chongqing Medical University(重庆医科大学第一附属医院)
Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025
Federal University of Goiás(戈亚斯联邦大学);Federal University of Rio Grande do Norte(北里奥格兰德联邦大学);Aeronautics Institute of Technology(航空技术研究所);Federal University of Mato Grosso(马托格罗索联邦大学)
本文面向 INTERSPEECH 2025 自然条件语音情感识别挑战赛,提出结合先进音频模型、韵律与频谱增强文本特征、F0 量化和集成学习的鲁棒系统,并验证图注意力网络融合有效性,测试集 Macro F1 达 39.79%。
EGSTalker: Real-Time Audio-Driven Talking Head Generation with Efficient Gaussian Deformation
Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center, School of Computer Science and Technology, Xinjiang University(新疆多模态智能处理与信息安全部门工程技术研究中心,计算机科学与技术学院,新疆大学);Department of Computer Science and Technology, Tsinghua University(计算机科学与技术系,清华大学);School of Electrical Engineering and Automation, Tianjin University of Technology(电气工程与自动化学院,天津理工大学)
HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
EPIC Lab, Shanghai Jiao Tong University(上海交通大学EPIC实验室);DAIL Tech(DAIL科技);Northeastern University(东北大学);Sichuan University(四川大学);Huazhong University of Science and Technology(华中科技大学)
LPIPS-AttnWav2Lip: Generic Audio-Driven lip synchronization for Talking Head Generation in the Wild
University of Science and Technology Beijing(北京科技大学);Xiaoduo Intelligent Technology (Beijing) Co., Ltd(小多智能技术(北京)有限公司);Department of Computer Science, Changzhi University(长治大学计算机科学系)
GRAM: Spatial general-purpose audio representation models for real-world applications
Donders Institute, Radboud University, Nijmegen, The Netherlands(多纳茨研究所,拉德堡德大学,尼姆韦根,荷兰);Mortimer B Zuckerman Institute, Columbia University, New York, United States(莫蒂默·B·祖克erman研究所,哥伦比亚大学,纽约,美国)
AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Alibaba Group(阿里巴巴集团);Nanjing University(南京大学)
Accelerating Audio Research with Robotic Dummy Heads
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Stony Brook University(纽约州立大学石溪分校);University of Illinois Chicago(伊利诺伊大学芝加哥分校);Discovery Partners Institute(发现伙伴研究所);Amazon Web Services(亚马逊云服务)
Audio-Plane: Audio Factorization Plane Gaussian Splatting for Real-Time Talking Head Synthesis
School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电子与电气工程学院);John A. Paulson School of Engineering and Applied Sciences at Harvard University(哈佛大学约翰·A·保罗森工程与应用科学学院);PhiGent Robotics(PhiGent机器人技术公司);Beijing National Research Center for Information Science and Technology (BNRist) and the Department of Automation, Tsinghua University(北京信息科学与技术国家研究中心(BNRist)和清华大学自动化系)
PGSTalker: Real-Time Audio-Driven Talking Head Generation via 3D Gaussian Splatting with Pixel-Aware Density Control
Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center(新疆多模态智能处理与信息安全工程技术创新研究中心);School of Computer Science and Technology, Xinjiang University(新疆大学计算机科学与技术学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);School of Electrical Engineering and Automation, Tianjin University of Technology(天津理工大学电气工程与自动化学院)
Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation
ReLER, CCAI, Zhejiang University(ReLER,中国人工智能学会,浙江大学);School of Cyber Science and Technology, Sun Yat-sen University(信息科学与技术学院,孙中山大学);Department of Statistics, University of Wisconsin–Madison(统计学系,威斯康星大学麦迪逊分校)