多模态大模型

GAIS: Frame-Level Gated Audio-Visual Integration with Semantic Variance-Scaled Perturbation for Text-Video Retrieval

Bowen Yang, Yun Cao, Chen He, Xiaosu Su

机构 * School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments 13 pages

Detect Any Sound: Open-Vocabulary Sound Event Detection with Multi-Modal Queries

Pengfei Cai, Yan Song, Qing Gu, Nan Jiang, Haoyu Song, Ian McLoughlin

机构 * University of Science and Technology of China（科学技术大学） ； Singapore Institute of Technology（新加坡理工学院） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

Comments Accepted by MM 2025

$\texttt{AVROBUSTBENCH}$: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time

Sarthak Kumar Maharana, Saksham Singh Kushwaha, Baoming Zhang, Adrian Rodriguez, Songtao Wei, Yapeng Tian, Yunhui Guo

机构 * The University of Texas at Dallas（德克萨斯大学达拉斯分校） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

Comments 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Track on Datasets and Benchmarks

M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models

Yejin Kwon, Taewoo Kang, Hyunsoo Yoon, Changouk Kim

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

Comments Submitted to LREC 2026. 11 pages, 5 figures

Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding

Yolo Yunlong Tang, Daiki Shimada, Jing Bi, Mingqian Feng, Hang Hua, Chenliang Xu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted to AAAI 2025

Unlocking Multimodal Mathematical Reasoning via Process Reward Model

Ruilin Luo, Zhuofan Zheng, Yifan Wang, Xinzhe Ni, Zicheng Lin, Songtao Jiang, Yiyao Yu, Chufan Shi, Lei Wang, Ruihang Chu, Jin Zeng, Yujiu Yang

机构 * Tsinghua University（清华大学） ； ByteDance（字节跳动） ； Zhejiang University（浙江大学） ； Ping An Technology (Shenzhen) Co., Ltd（平安科技（深圳）有限公司） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

Comments NeurIPS 2025 Main Track

Iterative Residual Cross-Attention Mechanism: An Integrated Approach for Audio-Visual Navigation Tasks

Hailong Zhang, Yinfeng Yu, Liejun Wang, Fuchun Sun, Wendong Zheng

机构 * Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center, School of Computer Science and Technology, Xinjiang University（新疆多模态智能处理与信息安全工程技术创新中心，计算机科学与技术学院，新疆大学） ； Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学） ； School of Electrical Engineering and Automation, Tianjin University of Technology（电气工程与自动化学院，天津工业大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM

Comments Accepted for publication by IEEE International Conference on Systems, Man, and Cybernetics 2025

MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech Paralinguistic and Affect Labeling

Yifan Cheng, Ruoyi Zhang, Jiatong Shi

机构 * Santa Clara, CA, USA（美国圣克拉拉） ； Carnegie Mellon University（卡内基梅隆大学） ； Huazhong University of Science and Technology（华中科技大学） ； Nanjing University of Information Science and Technology（南京信息工程大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CL、eess.AS

Comments Accepted by Interspeech

Journal ref Proc. of Interspeech2025

OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination

Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Chao Sun, Rongzhou Zhang, Guanyu Zhou, Lijie Wen, Xuming Hu

机构 * Tsinghua University（清华大学） ； The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）） ； OpenRL ； Chongqing University（重庆大学） 纠错

专题命中 音频语音多模态 ：omni-modal(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM

MM-HSD: Multi-Modal Hate Speech Detection in Videos

Berta Céspedes-Sarrias, Carlos Collado-Capell, Pablo Rodenas-Ruiz, Olena Hrynenko, Andrea Cavallaro

机构 * EPFL（苏黎世联邦理工学院） ； Idiap Research Institute（日内瓦研究所） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI、cs.MM

Comments Accepted at ACM Multimedia 2025

Human-Inspired Computing for Robust and Efficient Audio-Visual Speech Recognition

Qianhui Liu, Jiadong Wang, Yang Wang, Xin Yang, Gang Pan, Haizhou Li

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM、eess.AS

Comments aceepted by IEEE TC

Dopamine Audiobook: A Training-free MLLM Agent for Emotional and Immersive Audiobook Generation

Yan Rong, Shan Yang, Chenxing Li, Dong Yu, Li Liu

专题命中 音频语音多模态 ：MLLM(title,abstract)；multimodal(abstract)；分类 cs.MM、eess.AS

Multi-Agent Game Generation and Evaluation via Audio-Visual Recordings

Alexia Jolicoeur-Martineau

机构 * Samsung SAIL Montréal（三星SAIL蒙特利尔） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；omni-modal(abstract)；分类 cs.AI、cs.MM

Progressive Homeostatic and Plastic Prompt Tuning for Audio-Visual Multi-Task Incremental Learning

Jiong Yin, Liang Li, Jiehua Zhang, Yuhan Gao, Chenggang Yan, Xichun Sheng

机构 * Hangzhou Dianzi University（杭州电子科技大学） ； Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所） ； Xi’an Jiaotong University（西安交通大学） ； Macao Polytechnic University（澳门 polytechnic university） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICCV 2025

MST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge Distillation

Hui Li, Pengfei Yang, Juanyang Chen, Le Dong, Yanxin Chen, Quan Wang

机构 * Xidian University（西安电子科技大学） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

Comments Accepted to ACM MM 2025 (The 33rd ACM International Conference on Multimedia)

AVE Speech: A Comprehensive Multi-Modal Dataset for Speech Recognition Integrating Audio, Visual, and Electromyographic Signals

Dongliang Zhou, Yakun Zhang, Jinghan Wu, Xingyu Zhang, Liang Xie, Erwei Yin

机构 * Defense Innovation Institute, Academy of Military Sciences, Beijing, China（国防科技研究院，军事科学院，北京，中国） ； Tianjin Artificial Intelligence Innovation Center, Tianjin, China（天津人工智能创新中心，天津，中国） ； Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学，深圳，中国） ； Tianjin University, Tianjin, China（天津大学，天津，中国） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

Comments The paper has been accepted by IEEE Transactions on Human-Machine Systems

Leveraging Unlabeled Audio-Visual Data in Speech Emotion Recognition using Knowledge Distillation

Varsha Pendyala, Pedro Morgado, William Sethares

机构 * Department of Electrical and Computer Engineering（电气与计算机工程系） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.MM、eess.AS

Comments Accepted at INTERSPEECH 2025

WhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Prediction

Jakaria Islam Emon, Kazi Tamanna Alam, Md. Abu Salek

机构 * Hokkaido Denshikiki Co., Ltd.（Hokkaido Denshikiki公司） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.AI、eess.AS

Comments 3 pages

Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation

Weiqi Feng, Yangrui Chen, Shaoyu Wang, Yanghua Peng, Haibin Lin, Minlan Yu

机构 * Harvard University（哈佛大学） ； Bytedance（字节跳动） ； University of Southern California（南加州大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion

Shunian Chen, Xinyuan Xie, Zheshu Chen, Liyan Zhao, Owen Lee, Zhan Su, Qilin Sun, Benyou Wang

机构 * The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis

Minsu Kim, Pingchuan Ma, Honglie Chen, Stavros Petridis, Maja Pantic

机构 * Meta AIUK（Meta AI英国分公司） ； Imperial College London（伦敦帝国学院） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.AI、eess.AS

Comments Interspeech 2025

CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization

Detao Bai, Zhiheng Ma, Xihan Wei, Liefeng Bo

机构 * Tongyi Lab, Alibaba Group（通义实验室，阿里巴巴集团） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、eess.AS

UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing

Yung-Hsuan Lai, Janek Ebbers, Yu-Chiang Frank Wang, François Germain, Michael Jeffrey Jones, Moitreya Chatterjee

机构 * Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所） ； NVIDIA, Taiwan（台湾NVIDIA公司） ； Mitsubishi Electric Research Labs (MERL)（三菱电机研究实验室） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments CVPR 2025

mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition

Andrew Rouditchenko, Samuel Thomas, Hilde Kuehne, Rogerio Feris, James Glass

机构 * MIT（麻省理工学院） ； MIT-IBM Watson AI Lab（麻省理工-IBM沃森人工智能实验室） ； University of Tuebingen（图宾根大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments Accepted in Signal Processing Letters. Code at https://github.com/roudimit/whisper-flamingo

Tailored Design of Audio-Visual Speech Recognition Models using Branchformers

David Gimeno-Gómez, Carlos-D. Martínez-Hinarejos

机构 * Pattern Recognition and Human Language Technology research center, Universitat Politècnica de València, Spain（模式识别与人类语言技术研究中心，西班牙巴塞罗那理工大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted in Computer Speech & Language journal of Elsevier

Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation

Sungnyun Kim, Sungwoo Cho, Sangmin Bae, Kangwook Jang, Se-Young Yun

机构 * Kim Jaechul Graduate School of AI, KAIST（金jaechul人工智能研究生院，韩国科学技术院） ； School of Electrical Engineering, KAIST（电气工程学院，韩国科学技术院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

Comments ICLR 2025; 22 pages, 6 figures, 14 tables

Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides

Jinghua Zhao, Yuhang Jia, Shiyao Wang, Jiaming Zhou, Hui Wang, Yong Qin

机构 * College of Computer Science, Nankai University, Tianjin, China（南开大学计算机科学学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM

Comments 6 pages, 7 figures

MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted to CVPR 2025. Project page: https://hkchengrex.github.io/MMAudio

GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations

Yupei Li, Qiyang Sun, Sunil Munthumoduku Krishna Murthy, Emran Alturki, Björn W. Schuller

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generation

Haomin Zhang, Chang Liu, Junjie Zheng, Zihao Chen, Chaofan Ding, Xinhan Di

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments 11 pages, 5 figures