多模态大模型

Dynamic Content Moderation in Livestreams: Combining Supervised Classification with MLLM-Boosted Similarity Matching

直播中的动态内容审核：结合监督分类与MLLM增强的相似度匹配

Wei Chee Yew, Hailun Xu, Sanjay Saha, Xiaotian Fan, Hiok Hian Ong, David Yuchen Wang, Kanchan Sarkar, Zhenheng Yang, Danhui Guan

机构 * TikTok Singapore Singapore（TikTok新加坡） ； TikTok San Jose United States（TikTok旧金山美国） ； TikTok Shanghai China（TikTok上海中国） 纠错

专题命中 音频语音多模态 ：MLLM(title,title_cn)；multimodal(abstract)；分类 cs.CV、cs.AI

AI总结 提出一种混合审核框架，结合监督分类和基于参考的相似度匹配，利用多模态大语言模型提升准确性，在保持轻量推理的同时实现大规模直播内容审核。

Comments To be published at KDD 2026 (ADS track)

Music Audio-Visual Question Answering Requires Specialized Multimodal Designs

音乐音频视觉问答需要专门的多模态设计

Wenhao You, Xingjian Diao, Wenjun Huang, Chunhui Zhang, Keyi Kong, Weiyi Wu, Chiyu Ma, Zhongyu Ouyang, Tingxuan Wu, Ming Cheng, Soroush Vosoughi, Jiang Gui

机构 * University of Waterloo（滑铁卢大学） ； Dartmouth College（达特茅斯学院） ； UC Irvine（加州大学尔湾分校） ； New York University（纽约大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

AI总结 本文探讨了音乐音频视觉问答任务中多模态设计的必要性，指出需专门的输入处理、空间时间架构和音乐特定建模策略以应对连续密集的音频视觉内容和复杂时间动态。

Comments Accepted to Annual Meeting of the Association for Computational Linguistics (ACL 2026). The first two authors contributed equally

Integrating Fine-Grained Audio-Visual Evidence for Robust Multimodal Emotion Reasoning

融合细粒度音频视觉证据以实现鲁棒的多模态情绪推理

Zhixian Zhao, Wenjie Tian, Lei Xie

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

AI总结 SABER-LLM通过构建大规模情绪推理数据集和结构化证据分解范式，实现鲁棒的多模态情绪推理。

Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs

Umberto Cappellazzo, Minsu Kim, Stavros Petridis

机构 * Imperial College London（伦敦帝国学院） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to IEEE ASRU 2025

MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens

Jeong Hun Yeo, Hyeongseop Rha, Se Jin Park, Yong Man Ro

机构 * Integrated Vision and Language Lab, KAIST, South Korea（集成视觉与语言实验室，韩国科学技术院） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at Findings of ACL 2025. The code and models are available https://github.com/JeongHun0716/MMS-LLaMA

Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoising

Yan-Bo Lin, Kevin Lin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Chung-Ching Lin, Xiaofei Wang, Gedas Bertasius, Lijuan Wang

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Project page: https://genjib.github.io/project_page/AVED/index.html

Pilot-guided Multimodal Semantic Communication for Audio-Visual Event Localization

Fei Yu, Zhe Xiang, Nan Che, Zhuoran Zhang, Yuandi Li, Junxiao Xue, Zhiguo Wan

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

AlignVSR: Audio-Visual Cross-Modal Alignment for Visual Speech Recognition

Zehua Liu, Xiaolou Li, Chen Chen, Li Guo, Lantian Li, Dong Wang

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extraction

Zhaoxi Mu, Xinyu Yang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(title)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by IJCAI 2024

Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder

Yusheng Dai, Hang Chen, Jun Du, Xiaofei Ding, Ning Ding, Feijun Jiang, Chin-Hui Lee

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 6 pages, 2 figures, published in ICME2023

Multi-Modal Multi-Correlation Learning for Audio-Visual Speech Separation

Xiaoyu Wang, Xiangyu Kong, Xiulian Peng, Yan Lu

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 5 pages, accepted by interspeech2022

MPN: Multimodal Parallel Network for Audio-Visual Event Localization

Jiashuo Yu, Ying Cheng, Rui Feng

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments IEEE International Conference on Multimedia and Expo (ICME) 2021 Oral

Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videos

Kranti Kumar Parida, Neeraj Matiyali, Tanaya Guha, Gaurav Sharma

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments To appear in WACV 2020, Project Page: https://cse.iitk.ac.in/users/kranti/avzsl.html

Benchmarking Commercial Speech Recognition and Multimodal Large Language Models on Dysarthric Speech: Severity-Stratified Baselines and Architecture-Specific Prompting Effects

基于商业自动语音识别和多模态大语言模型的口吃语音零样本识别

Ali Alsayegh, Tariq Masood

专题命中 音频语音多模态 ：MLLM(summary_cn,abstract)；multimodal(title,abstract)；分类 eess.AS

AI总结 本研究评估了商业ASR和MLLM在口吃语音识别中的性能，发现GPT-4o在重度口吃情况下显著降低WER，而Gemini变体表现下降，为辅助语音接口技术选择提供实证依据。

Comments 32 pages. Revised peer-reviewed version. Updated dataset filtering and transcript normalization, expanded four-condition prompting ablation and error analysis, and revised statistical reporting. Published in International Journal of Intelligent Systems

Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QA

Qwen-MusicAVQA-7B：一种用于音乐音频-视觉问答的多模态模型

Maryam Dehdashti

机构 * Inference Matter Labs（推理物质实验室） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(title)；omni-modal(abstract)；分类 cs.CV、cs.MM

AI总结 该研究提出轻量级多模态模型Qwen-MusicAVQA-7B，通过连接冻结的Whisper编码器与Qwen2-VL-7B-Instruct，在MUSIC-AVQA等基准上实现高准确率，训练成本低，且发现音频时间信息保留程度影响下游问答准确率。

Comments 24 pages, 1 figure, 8 tables. Code: https://github.com/MKDehdashti/Qwen2-vl-audio Checkpoints: https://huggingface.co/MayaKD/qwen2-vl-audio

Beyond Generative Decoding: Discriminative Hidden-State Readout from a Native Omni-Modal LLM for Multimodal Sentiment Analysis

超越生成式解码：来自原生全模态大语言模型的判别性隐藏状态读出用于多模态情感分析

Bin Wen, Tien-Ping Tan

机构 * School of Computer Sciences, Universiti Sains Malaysia, Penang, Malaysia（计算机科学学院，马来西亚国际科学大学，槟城） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；omni-modal(title,abstract)；分类 cs.MM、eess.AS

AI总结 针对多模态情感分析中生成式读出将连续回归绑定到离散自回归解码导致精度和效率损失的问题，提出基于原生全模态大语言模型Qwen2.5-Omni-7B的Thinker模块的判别性读出方法，通过轻量回归头直接映射最终层隐藏状态，在单消费级GPU上实现最先进性能。

Comments 18 pages, 4 figures, 6 tables

Probing Cross-modal Information Hubs in Audio-Visual LLMs

探测音频-视觉大语言模型中的跨模态信息枢纽

Jihoo Jung, Chaeyoung Jung, Ji-Hoon Kim, Joon Son Chung

机构 * Department of Electrical Engineering, Korea Advanced Institute of Science ； The Graduate School of Advanced Imaging Science, Multimedia \& Film, Chung-Ang University, Seoul, Republic of Korea 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.AI、eess.AS

AI总结 本文研究了音频-视觉大语言模型中音频与视觉模态间的跨模态信息流动，发现信息主要存储在sink tokens中，并提出一种无需训练的hallucination缓解方法。

Comments Accepted by ICML 2026

Evaluating Hallucinations in Audio-Visual Multimodal LLMs with Spoken Queries under Diverse Acoustic Conditions

通过多样声学条件下的语音查询评估音频-视觉多模态大语言模型的幻觉

Hansol Park, Hoseong Ahn, Junwon Moon, Yejin Lee, Kyuhong Shim

机构 * Department of Intelligent Software, Sungkyunkwan University（智能软件系，成均馆大学） ； Department of Computer Science and Engineering, Sungkyunkwan University（计算机科学与工程系，成均馆大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title)；image-text(abstract)；分类 cs.AI、eess.AS

AI总结 研究探讨语音查询对多模态幻觉的影响，通过RePOPE-Spk基准测试发现语音查询导致错误率显著增加，提出改进方法以提升语音界面可靠性。

Comments Submitted to Interspeech2026

A Survey on Cross-Modal Interaction Between Music and Multimodal Data

多模态数据与音乐交叉交互的综述

Sifei Li, Mining Tan, Feier Shen, Minyan Luo, Zijiao Yin, Fan Tang, Weiming Dong, Changsheng Xu

机构 * MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China. E-mail: S. Li ； School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100190, China. E-mail: F. Shen ； Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China. E-mail: . 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(title,abstract)；分类 cs.MM、eess.AS

AI总结 本文综述了音乐与多模态数据的交叉交互，探讨了音乐在多模态学习中的作用，并提出了未来研究的方向。

Comments 34 pages, 7 figures

Session-Level Spoken Language Assessment with a Multimodal Foundation Model via Multi-Target Learning

Hong-Yun Lin, Jhen-Ke Lin, Chung-Chun Wang, Hao-Chien Lu, Berlin Chen

机构 * Department of Computer Science（计算机科学系） ； Information Engineering, National Taiwan Normal University（信息工程，台湾正常大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；multimodal foundation model(title,abstract)；分类 cs.CL、cs.AI

Comments Copyright 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

Teacher-Guided Pseudo Supervision and Cross-Modal Alignment for Audio-Visual Video Parsing

Yaru Chen, Ruohao Guo, Liting Gao, Yang Xiang, Qingyu Luo, Zhenbo Li, Wenwu Wang

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM

Multi-source Multimodal Progressive Domain Adaption for Audio-Visual Deception Detection

Ronghao Lin, Sijie Mai, Ying Zeng, Qiaolin He, Aolin Xiong, Haifeng Hu

机构 * Sun Yat-Sen University（中山大学） ； Nanyang Technological University（南洋理工大学） ； South China Normal University（华南师范大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM

Comments Accepted at ACM MM 2025 SVC Workshop

EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning

Zhenghao Xing, Xiaowei Hu, Chi-Wing Fu, Wenhai Wang, Jifeng Dai, Pheng-Ann Heng

机构 * The Chinese University of Hong Kong（香港中文大学） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； Tsinghua University（清华大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title)；cross-modal(abstract)；分类 cs.CV、eess.AS

Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Models

Jinming Wen, Xinyi Wu, Shuai Zhao, Yanhao Jia, Yuwen Li

机构 * Jilin University（吉林大学） ； Shanghai Jiao Tong University（上海交通大学） ； Nanyang Technological University（南洋理工大学） ； Northeastern University（东北大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.AI、cs.MM

Lightweight Joint Audio-Visual Deepfake Detection via Single-Stream Multi-Modal Learning Framework

Kuiyuan Zhang, Wenjie Pei, Rushi Lan, Yifang Guo, Zhongyun Hua

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.AI、eess.AS

Towards Source Attribution of Singing Voice Deepfake with Multimodal Foundation Models

Orchid Chetia Phukan, Girish, Mohd Mujtaba Akhtar, Swarup Ranjan Behera, Priyabrata Mallick, Pailla Balakrishna Reddy, Arun Balaji Buduru, Rajesh Sharma

机构 * IIIT-DelhiIndia（印度IIIT-Delhi） ； UPESIndia（印度UPES） ； V.B.S.P.UIndia（印度V.B.S.P.U） ； Independent ResearcherIndia（印度独立研究者） ； Reliance AIIndia（印度Reliance AI） ； University of TartuEstonia（爱沙尼亚塔尔图大学） ； Plaksha UniversityIndia（印度Plaksha大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；multimodal foundation model(title,abstract)；分类 cs.MM、eess.AS

Comments Accepted to INTERSPEECH 2025

The Multimodal Information Based Speech Processing (MISP) 2025 Challenge: Audio-Visual Diarization and Recognition

Ming Gao, Shilong Wu, Hang Chen, Jun Du, Chin-Hui Lee, Shinji Watanabe, Jingdong Chen, Siniscalchi Sabato Marco, Odette Scharenborg

机构 * University of Science and Technology of China（中国科学技术大学） ； Georgia Institute of Technology（佐治亚理工学院） ； Carnegie Mellon University（卡内基梅隆大学） ； Northwestern Polytechnical University（西北工业大学） ； University of Palermo（巴勒莫大学） ； Delft University of Technology（代尔夫特理工大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(title)；multi-modal(abstract)；分类 cs.AI、eess.AS

Comments Accepted by Interspeech 2025. Camera-ready version

Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing

Pengcheng Zhao, Jinxing Zhou, Yang Zhao, Dan Guo, Yanxiang Chen

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(title)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by AAAI-2025

Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition

Sungnyun Kim, Kangwook Jang, Sangmin Bae, Hoirin Kim, Se-Young Yun

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted at SLT 2024 Main Conference; Code is available at https://github.com/sungnyun/avsr-temporal-dynamics

A multi-modal approach for identifying schizophrenia using cross-modal attention

Gowtham Premananth, Yashish M. Siriwardena, Philip Resnik, Carol Espy-Wilson

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(title,abstract)；分类 cs.MM、eess.AS

Comments Accepted to Annual International Conference of the IEEE Engineering in Medicine and Biology Society 2024