多模态大模型

Iterative Residual Cross-Attention Mechanism: An Integrated Approach for Audio-Visual Navigation Tasks

Hailong Zhang, Yinfeng Yu, Liejun Wang, Fuchun Sun, Wendong Zheng

机构 * Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center, School of Computer Science and Technology, Xinjiang University（新疆多模态智能处理与信息安全工程技术创新中心，计算机科学与技术学院，新疆大学） ； Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学） ； School of Electrical Engineering and Automation, Tianjin University of Technology（电气工程与自动化学院，天津工业大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM

Comments Accepted for publication by IEEE International Conference on Systems, Man, and Cybernetics 2025

MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech Paralinguistic and Affect Labeling

Yifan Cheng, Ruoyi Zhang, Jiatong Shi

机构 * Santa Clara, CA, USA（美国圣克拉拉） ； Carnegie Mellon University（卡内基梅隆大学） ； Huazhong University of Science and Technology（华中科技大学） ； Nanjing University of Information Science and Technology（南京信息工程大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CL、eess.AS

Comments Accepted by Interspeech

Journal ref Proc. of Interspeech2025

OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination

Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Chao Sun, Rongzhou Zhang, Guanyu Zhou, Lijie Wen, Xuming Hu

机构 * Tsinghua University（清华大学） ； The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）） ； OpenRL ； Chongqing University（重庆大学） 纠错

专题命中 音频语音多模态 ：omni-modal(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM

MM-HSD: Multi-Modal Hate Speech Detection in Videos

Berta Céspedes-Sarrias, Carlos Collado-Capell, Pablo Rodenas-Ruiz, Olena Hrynenko, Andrea Cavallaro

机构 * EPFL（苏黎世联邦理工学院） ； Idiap Research Institute（日内瓦研究所） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI、cs.MM

Comments Accepted at ACM Multimedia 2025

Human-Inspired Computing for Robust and Efficient Audio-Visual Speech Recognition

Qianhui Liu, Jiadong Wang, Yang Wang, Xin Yang, Gang Pan, Haizhou Li

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM、eess.AS

Comments aceepted by IEEE TC

Dopamine Audiobook: A Training-free MLLM Agent for Emotional and Immersive Audiobook Generation

Yan Rong, Shan Yang, Chenxing Li, Dong Yu, Li Liu

专题命中 音频语音多模态 ：MLLM(title,abstract)；multimodal(abstract)；分类 cs.MM、eess.AS

Multi-Agent Game Generation and Evaluation via Audio-Visual Recordings

Alexia Jolicoeur-Martineau

机构 * Samsung SAIL Montréal（三星SAIL蒙特利尔） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；omni-modal(abstract)；分类 cs.AI、cs.MM

Progressive Homeostatic and Plastic Prompt Tuning for Audio-Visual Multi-Task Incremental Learning

Jiong Yin, Liang Li, Jiehua Zhang, Yuhan Gao, Chenggang Yan, Xichun Sheng

机构 * Hangzhou Dianzi University（杭州电子科技大学） ； Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所） ； Xi’an Jiaotong University（西安交通大学） ； Macao Polytechnic University（澳门 polytechnic university） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICCV 2025

MST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge Distillation

Hui Li, Pengfei Yang, Juanyang Chen, Le Dong, Yanxin Chen, Quan Wang

机构 * Xidian University（西安电子科技大学） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

Comments Accepted to ACM MM 2025 (The 33rd ACM International Conference on Multimedia)

AVE Speech: A Comprehensive Multi-Modal Dataset for Speech Recognition Integrating Audio, Visual, and Electromyographic Signals

Dongliang Zhou, Yakun Zhang, Jinghan Wu, Xingyu Zhang, Liang Xie, Erwei Yin

机构 * Defense Innovation Institute, Academy of Military Sciences, Beijing, China（国防科技研究院，军事科学院，北京，中国） ； Tianjin Artificial Intelligence Innovation Center, Tianjin, China（天津人工智能创新中心，天津，中国） ； Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学，深圳，中国） ； Tianjin University, Tianjin, China（天津大学，天津，中国） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

Comments The paper has been accepted by IEEE Transactions on Human-Machine Systems

Leveraging Unlabeled Audio-Visual Data in Speech Emotion Recognition using Knowledge Distillation

Varsha Pendyala, Pedro Morgado, William Sethares

机构 * Department of Electrical and Computer Engineering（电气与计算机工程系） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.MM、eess.AS

Comments Accepted at INTERSPEECH 2025

WhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Prediction

Jakaria Islam Emon, Kazi Tamanna Alam, Md. Abu Salek

机构 * Hokkaido Denshikiki Co., Ltd.（Hokkaido Denshikiki公司） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.AI、eess.AS

Comments 3 pages

Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation

Weiqi Feng, Yangrui Chen, Shaoyu Wang, Yanghua Peng, Haibin Lin, Minlan Yu

机构 * Harvard University（哈佛大学） ； Bytedance（字节跳动） ； University of Southern California（南加州大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion

Shunian Chen, Xinyuan Xie, Zheshu Chen, Liyan Zhao, Owen Lee, Zhan Su, Qilin Sun, Benyou Wang

机构 * The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis

Minsu Kim, Pingchuan Ma, Honglie Chen, Stavros Petridis, Maja Pantic

机构 * Meta AIUK（Meta AI英国分公司） ； Imperial College London（伦敦帝国学院） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.AI、eess.AS

Comments Interspeech 2025

CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization

Detao Bai, Zhiheng Ma, Xihan Wei, Liefeng Bo

机构 * Tongyi Lab, Alibaba Group（通义实验室，阿里巴巴集团） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、eess.AS

UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing

Yung-Hsuan Lai, Janek Ebbers, Yu-Chiang Frank Wang, François Germain, Michael Jeffrey Jones, Moitreya Chatterjee

机构 * Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所） ； NVIDIA, Taiwan（台湾NVIDIA公司） ； Mitsubishi Electric Research Labs (MERL)（三菱电机研究实验室） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments CVPR 2025

mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition

Andrew Rouditchenko, Samuel Thomas, Hilde Kuehne, Rogerio Feris, James Glass

机构 * MIT（麻省理工学院） ； MIT-IBM Watson AI Lab（麻省理工-IBM沃森人工智能实验室） ； University of Tuebingen（图宾根大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments Accepted in Signal Processing Letters. Code at https://github.com/roudimit/whisper-flamingo

Tailored Design of Audio-Visual Speech Recognition Models using Branchformers

David Gimeno-Gómez, Carlos-D. Martínez-Hinarejos

机构 * Pattern Recognition and Human Language Technology research center, Universitat Politècnica de València, Spain（模式识别与人类语言技术研究中心，西班牙巴塞罗那理工大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted in Computer Speech & Language journal of Elsevier

Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation

Sungnyun Kim, Sungwoo Cho, Sangmin Bae, Kangwook Jang, Se-Young Yun

机构 * Kim Jaechul Graduate School of AI, KAIST（金jaechul人工智能研究生院，韩国科学技术院） ； School of Electrical Engineering, KAIST（电气工程学院，韩国科学技术院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

Comments ICLR 2025; 22 pages, 6 figures, 14 tables

Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides

Jinghua Zhao, Yuhang Jia, Shiyao Wang, Jiaming Zhou, Hui Wang, Yong Qin

机构 * College of Computer Science, Nankai University, Tianjin, China（南开大学计算机科学学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM

Comments 6 pages, 7 figures

MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted to CVPR 2025. Project page: https://hkchengrex.github.io/MMAudio

GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations

Yupei Li, Qiyang Sun, Sunil Munthumoduku Krishna Murthy, Emran Alturki, Björn W. Schuller

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generation

Haomin Zhang, Chang Liu, Junjie Zheng, Zihao Chen, Chaofan Ding, Xinhan Di

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments 11 pages, 5 figures

Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions

Hadi Amini, Md Jueal Mia, Yasaman Saadati, Ahmed Imteaj, Seyedsina Nabavirazavi, Urmish Thakker, Md Zarif Hossain, Awal Ahmed Fime, S. S. Iyengar

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

Towards Open-Vocabulary Audio-Visual Event Localization

Jinxing Zhou, Dan Guo, Ruohao Guo, Yuxin Mao, Jingjing Hu, Yiran Zhong, Xiaojun Chang, Meng Wang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

Comments accepted by CVPR 2025; Project page: https://github.com/jasongief/OV-AVEL

Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixup

Seokun Kang, Taehwan Kim

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Language Model Mapping in Multimodal Music Learning: A Grand Challenge Proposal

Daniel Chin, Gus Xia

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

Global-Local Distillation Network-Based Audio-Visual Speaker Tracking with Incomplete Modalities

Yidi Li, Yihan Li, Yixin Guo, Bin Ren, Zhenhuan Xu, Hao Guo, Hong Liu, Nicu Sebe

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments We request to withdraw our paper from arXiv due to unresolved author disagreements about the data interpretation and study conclusions. To maintain scientific integrity, we believe withdrawing the paper is necessary. We regret any confusion caused

LSTMSE-Net: Long Short Term Speech Enhancement Network for Audio-visual Speech Enhancement

Arnav Jain, Jasmer Singh Sanjotra, Harshvardhan Choudhary, Krish Agrawal, Rupal Shah, Rohan Jha, M. Sajid, Amir Hussain, M. Tanveer

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.MM、eess.AS

Journal ref INTERSPEECH 2024