多模态大模型 - arXivDaily 专题

2210.15563 2024-03-19 cs.CV cs.IR cs.SD eess.AS 88%

Multimodal Transformer Distillation for Audio-Visual Synchronization

Xuanjun Chen, Haibin Wu, Chung-Che Wang, Hung-yi Lee, Jyh-Shing Roger Jang

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted by ICASSP 2024

2401.12264 2024-02-22 eess.AS cs.MM cs.SD eess.IV 88%

CoAVT: A Cognition-Inspired Unified Audio-Visual-Text Pre-Training Model for Multimodal Processing

Xianghu Yue, Xiaohai Tian, Lu Lu, Malu Zhang, Zhizheng Wu, Haizhou Li

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM、eess.AS

2311.12826 2023-11-23 cs.CV cs.AI 88%

LiveChat: Video Comment Generation from Audio-Visual Multimodal Contexts

Julien Lalanne, Raphael Bournet, Yi Yu

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.AI

2310.07517 2023-10-12 cs.CV cs.MM 88%

CM-PIE: Cross-modal perception for interactive-enhanced audio-visual video parsing

Yaru Chen, Ruohao Guo, Xubo Liu, Peipei Wu, Guangyao Li, Zhenbo Li, Wenwu Wang

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM

Comments 5 pages, 3 figures, 15 references

2303.05338 2023-03-14 cs.SD cs.MM eess.AS 88%

MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning

Ruize Xu, Ruoxuan Feng, Shi-Xiong Zhang, Di Hu

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM、eess.AS

2302.11254 2023-02-23 cs.SD cs.CV cs.LG eess.AS eess.IV 88%

Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification

Meng Liu, Kong Aik Lee, Longbiao Wang, Hanyi Zhang, Chang Zeng, Jianwu Dang

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

2212.00539 2022-12-02 cs.MM cs.CV 88%

Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection

Rahul Sharma, Shrikanth Narayanan

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM

Comments Under review at OJSP

2211.03434 2022-11-08 cs.MM cs.AI 88%

Complete Cross-triplet Loss in Label Space for Audio-visual Cross-modal Retrieval

Donghuo Zeng, Yanan Wang, Jianming Wu, Kazushi Ikeda

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.AI、cs.MM

Comments 9 pages, 5 figures, 3 tables, accepted by IEEE ISM 2022

1709.00944 2022-04-19 cs.SD cs.MM eess.AS stat.ML 88%

Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks

Jen-Cheng Hou, Syu-Siang Wang, Ying-Hui Lai, Yu Tsao, Hsiu-Wen Chang, Hsin-Min Wang

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM、eess.AS

Comments This paper is the same as arXiv:1703.10893v6. Apologies for the inconvenience. arXiv admin note: text overlap with arXiv:1703.10893

2201.02184 2022-03-15 eess.AS cs.CV cs.SD 88%

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman Mohamed

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments ICLR 2022

2104.04598 2021-06-22 cs.SD cs.CV cs.LG eess.AS eess.IV 88%

Cross-Modal learning for Audio-Visual Video Parsing

Jatin Lamba, Abhishek, Jayaprakash Akula, Rishabh Dabral, Preethi Jyothi, Ganesh Ramakrishnan

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Work accepted at Interspeech 2021

2104.02775 2021-04-08 cs.CV cs.SD eess.AS eess.IV 88%

Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation

Jiyoung Lee, Soo-Whan Chung, Sunok Kim, Hong-Goo Kang, Kwanghoon Sohn

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments CVPR 2021. The first two authors contributed equally to this work. Project page: https://caffnet.github.io

2104.02000 2021-04-06 cs.CV cs.CR cs.SD eess.AS 88%

Can audio-visual integration strengthen robustness under multimodal attacks?

Yapeng Tian, Chenliang Xu

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments CVPR 2021

2102.03424 2021-02-16 cs.CV cs.SD eess.AS eess.IV 88%

Learning Audio-Visual Correlations from Variational Cross-Modal Generation

Ye Zhu, Yu Wu, Hugo Latapie, Yi Yang, Yan Yan

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted to ICASSP 2021

2005.13402 2020-11-24 cs.CV cs.SD eess.AS 88%

AVGZSLNet: Audio-Visual Generalized Zero-Shot Learning by Reconstructing Label Features from Multi-Modal Embeddings

Pratik Mazumder, Pravendra Singh, Kranti Kumar Parida, Vinay P. Namboodiri

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(title)；cross-modal(abstract)；分类 cs.CV、eess.AS

Comments Accepted in WACV 2021

2409.12319 2025-03-10 cs.CV cs.MM cs.SD eess.AS 88%

Large Language Models are Strong Audio-Visual Speech Recognition Learners

Umberto Cappellazzo, Minsu Kim, Honglie Chen, Pingchuan Ma, Stavros Petridis, Daniele Falavigna, Alessio Brutti, Maja Pantic

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；multi-modal(abstract)；MLLM(abstract)

Comments Accepted for publication at ICASSP 2025. The code and checkpoints are available here: https://github.com/umbertocappellazzo/Llama-AVSR

2608.15522 2026-08-18 cs.CV 新提交 88%

Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention

基于同步感知跨模态稀疏注意力的高效视听生成

Shengchuan Gao, Teng Hu, Bohao Feng, Luchen Li, Wenqiang Wang, Hongqian Deng, Ran Yi

机构 * Alibaba Group（阿里巴巴集团）； Alibaba Cloud Computing（阿里巴巴云计算）； Shanghai Jiao Tong University（上海交通大学）

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

AI总结本文提出同步感知加速框架，通过受保护的稀疏注意力策略在保留视听生成质量与同步性的同时提升推理效率。

2606.08231 2026-06-09 cs.CV 新提交 88%

Test-Time Scaling in Multimodal Foundation Models: A Comprehensive Survey of Generation and Reasoning

多模态基础模型中的测试时扩展：生成与推理的综合调查

Cong Wan, Ying He, Zhongzhan Huang, Hefeng Wu

机构 * Sun Yat-sen University（中山大学）

专题命中音频语音多模态：multimodal(title,abstract)；multimodal foundation model(title,abstract)；分类 cs.CV

AI总结本文首次系统综述多模态基础模型中的测试时扩展（TTS）方法，提出统一分类框架（采样、反馈、搜索三类），总结应用与基准，并讨论未来方向。

Comments Accepted by ACL 2026, Findings

2605.09906 2026-05-12 cs.AI cs.SD 88%

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

先分后融：通过模态特定的推理链减轻音频视觉大语言模型中的跨模态干扰

Xuanchen Li, Yuheng Lu, Chenrui Cui, Tianrui Wang, Zikang Huang, Yu Jiang, Long Zhou, Longbiao Wang, Jianwu Dang

机构 * Tianjin Key Laboratory of Cognitive Computing（天津认知计算实验室）； Tianjin University（天津大学）； Huiyan Technology Company, Ltd.（慧颜科技有限公司）； Chinese Academy of Sciences（中国科学院）； Tencent（腾讯）

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.AI

AI总结本文提出SFFL框架，通过模态特定的推理链减少跨模态干扰，提升音频视觉问答的准确性和鲁棒性，实验显示在通用AVQA基准和跨模态幻觉基准上分别提升5.16%和11.17%。

2512.03034 2026-03-10 cs.CV 88%

MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generation

MAViD：一种多模态框架用于音频-视觉对话理解和生成

Youxin Pang, Jiajun Liu, Lingfeng Tan, Yong Zhang, Feng Gao, Xiang Deng, Zhuoliang Kang, Xiaoming Wei, Yebin Liu

机构 * Tsinghua University（清华大学）； Meituan（美团）

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

AI总结 MAViD提出了一种多模态框架，通过Conductor-Creator架构实现音频-视觉对话的理解与生成，结合自回归和扩散模型提升长时多模态内容生成质量。

2602.08309 2026-02-10 cs.CV 88%

CAE-AV: Improving Audio-Visual Learning via Cross-modal Interactive Enrichment

CAE-AV：通过跨模态交互增强改进音频-视觉学习

Yunzuo Hu, Wen Li, Jing Zhang

机构 * School of Information Science and Engineering, East China University of Science and Technology (ECUST)（信息科学与工程学院，东华大学）

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

AI总结 CAE-AV通过跨模态交互增强框架，解决音频-视觉学习中的模态不对齐问题，提升表示质量和任务性能。

Comments 13 pages, 8 figures

2602.08293 2026-02-10 eess.AS 88%

Cross-Modal Bottleneck Fusion For Noise Robust Audio-Visual Speech Recognition

跨模态瓶颈融合用于噪声鲁棒的音频视觉语音识别

Seaone Ok, Min Jun Choi, Eungbeom Kim, Seungu Han, Kyogu Lee

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

AI总结 CoBRA通过跨模态瓶颈融合机制，在噪声环境下提升音频视觉语音识别的鲁棒性和效率。

Comments 5 pages, 3 figures, ICASSP 2026 Accepted

2508.01659 2026-01-27 cs.SD eess.AS 88%

From Contrast to Commonality: Audio Commonality Captioning for Enhanced Audio-Text Cross-modal Understanding in Multimodal LLMs

从对比到共同性：用于增强多模态大语言模型中音频-文本跨模态理解的音频共同性描述

Yuhang Jia, Xu Zhang, Yujie Guo, Yang Chen, Shiwan Zhao

机构 * TMCC, College of Computer Science, Nankai University, Tianjin, China（TMCC，计算机科学学院，南开大学，天津，中国）

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(title,abstract)；分类 eess.AS

AI总结本文提出音频共同性描述方法，旨在增强多模态大语言模型中音频与文本的跨模态理解，通过捕捉音频片段间的共享语义以改善模型的泛化能力与任务特定性能。

2503.21699 2025-12-09 cs.MM cs.AI cs.CV cs.SD eess.AS 88%

MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeX

MAVERIX：多模态音频视觉评估与识别指数

Liuyue Xie, Avik Kuthiala, George Z. Wei, Ce Zheng, Ananya Bal, Mosam Dabhi, Liting Wen, Taru Rustagi, Ethan Lai, Sushil Khyalia, Rohan Choudhury, Morteza Ziyadi, Xu Zhang, Hao Yang, László A. Jeni

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title)；分类 cs.CV、cs.AI、cs.MM

AI总结 MAVERIX是一个用于评估多模态模型音频视觉整合能力的统一基准，通过精心设计的问题展示模型在跨模态理解上的性能，揭示了与人类水平的显著差距。

Journal ref AAAI 2026

2508.17282 2025-12-04 cs.AI cs.SD 88%

ERF-BA-TFD+: A Multimodal Model for Audio-Visual Deepfake Detection

ERF-BA-TFD+: 一种用于音频视觉深度伪造检测的多模态模型

Xin Zhang, Jiaming Chu, Jian Zhao, Yuchu Jiang, Xu Yang, Lei Jin, Chi Zhang, Xuelong Li

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.AI

AI总结 ERF-BA-TFD+通过结合增强接收场和音频视觉融合，提出了一种多模态深度伪造检测模型，在DDL-AV数据集上实现了最先进的检测性能。

Comments The paper is withdrawn after discovering a flaw in the theoretical derivation presented in Section Method. The incorrect step leads to conclusions that are not supported by the corrected derivation. We plan to reconstruct the argument and will release an updated version once the issue is fully resolved

2408.14595 2025-11-19 cs.CL 88%

Surprisingly Fragile: Assessing and Addressing Prompt Instability in Multimodal Foundation Models

Ian Stewart, Sameera Horawalavithana, Brendan Kennedy, Sai Munikoti, Karl Pazdernik

机构 * Pacific Northwest National Laboratory（太平洋西北国家实验室）

专题命中音频语音多模态：multimodal(title,abstract)；multimodal foundation model(title,abstract)；分类 cs.CL

Comments arxiv

2511.09552 2025-11-13 cs.CR cs.MM 88%

Intelligent Carrier Allocation: A Cross-Modal Reasoning Framework for Adaptive Multimodal Steganography

Abhirup Das, Pranav Dudani, Shruti Sharma, Ravi Kumar C.

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(title,abstract)；分类 cs.MM

Comments 8 pages, 10 figures

2505.20873 2025-10-01 cs.CV 88%

Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models

Chaeyoung Jung, Youngjoon Jang, Jongmin Choi, Joon Son Chung

机构 * Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

2508.12334 2025-09-30 cs.SD cs.MM 88%

HDA-SELD: Hierarchical Cross-Modal Distillation with Multi-Level Data Augmentation for Low-Resource Audio-Visual Sound Event Localization and Detection

Qing Wang, Ya Jiang, Hang Chen, Sabato Marco Siniscalchi, Jun Du, Jianqing Gao

机构 * University of Science and Technology of China（中国科学技术大学）； University of Palermo（巴勒莫大学）； iFLYTEK Research（iFLYTEK研究院）

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM

Comments 13 pages, 8 figures

2506.23196 2025-07-01 cs.CV 88%

DEL: Dense Event Localization for Multi-modal Audio-Visual Understanding

Mona Ahmadian, Amir Shirian, Frank Guerin, Andrew Gilbert

机构 * University of Surrey（塞拉利昂大学）

专题命中音频语音多模态：multi-modal(title)；audio-visual(title)；multimodal(abstract)；cross-modal(abstract)

1. 音频语音多模态 4567 篇

Multimodal Transformer Distillation for Audio-Visual Synchronization

CoAVT: A Cognition-Inspired Unified Audio-Visual-Text Pre-Training Model for Multimodal Processing

LiveChat: Video Comment Generation from Audio-Visual Multimodal Contexts

CM-PIE: Cross-modal perception for interactive-enhanced audio-visual video parsing

MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning

Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification

Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection

Complete Cross-triplet Loss in Label Space for Audio-visual Cross-modal Retrieval

Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

Cross-Modal learning for Audio-Visual Video Parsing

Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation

Can audio-visual integration strengthen robustness under multimodal attacks?

Learning Audio-Visual Correlations from Variational Cross-Modal Generation

AVGZSLNet: Audio-Visual Generalized Zero-Shot Learning by Reconstructing Label Features from Multi-Modal Embeddings

Large Language Models are Strong Audio-Visual Speech Recognition Learners

Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention

Test-Time Scaling in Multimodal Foundation Models: A Comprehensive Survey of Generation and Reasoning

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generation

CAE-AV: Improving Audio-Visual Learning via Cross-modal Interactive Enrichment

Cross-Modal Bottleneck Fusion For Noise Robust Audio-Visual Speech Recognition

From Contrast to Commonality: Audio Commonality Captioning for Enhanced Audio-Text Cross-modal Understanding in Multimodal LLMs

MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeX

ERF-BA-TFD+: A Multimodal Model for Audio-Visual Deepfake Detection

Surprisingly Fragile: Assessing and Addressing Prompt Instability in Multimodal Foundation Models

Intelligent Carrier Allocation: A Cross-Modal Reasoning Framework for Adaptive Multimodal Steganography

Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models

HDA-SELD: Hierarchical Cross-Modal Distillation with Multi-Level Data Augmentation for Low-Resource Audio-Visual Sound Event Localization and Detection

DEL: Dense Event Localization for Multi-modal Audio-Visual Understanding