多模态信息融合 - arXivDaily 专题

2406.13384 2024-06-21 cs.SD cs.CV cs.MM eess.AS 62%

Straight Through Gumbel Softmax Estimator based Bimodal Neural Architecture Search for Audio-Visual Deepfake Detection

Aravinda Reddy PN, Raghavendra Ramachandra, Krothapalli Sreenivasa Rao, Pabitra Mitra, Vinod Rathod

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、cs.MM

2403.18715 2024-06-06 cs.CV cs.AI cs.CL cs.MM 62%

Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding

Xintong Wang, Jingheng Pan, Liang Ding, Chris Biemann

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments Accepted to Findings of ACL 2024

2402.17187 2024-04-18 eess.IV cs.CV 62%

PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction

Zhaoxin Guo, Zhipeng Wang, Ruiquan Ge, Jianxun Yu, Feiwei Qin, Yuan Tian, Yuqing Peng, Yonghong Li, Changmiao Wang

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、eess.IV

2401.04154 2024-01-10 cs.CV cs.AI cs.LG cs.MM cs.SD eess.AS 62%

Efficient Selective Audio Masked Multimodal Bottleneck Transformer for Audio-Video Classification

Wentao Zhu

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments Accepted by WACV 2024; well-formatted PDF is in https://drive.google.com/file/d/1qvW52lamsvNGMCqPS7q8g8L4NaR_LlbR/view?usp=sharing. arXiv admin note: text overlap with arXiv:2401.04023

2210.04135 2023-10-31 cs.CV cs.LG cs.MM 62%

VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment

Shraman Pramanick, Li Jing, Sayan Nag, Jiachen Zhu, Hardik Shah, Yann LeCun, Rama Chellappa

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；分类 cs.CV、cs.MM

Comments Published in TMLR 2023

2305.11019 2023-10-10 cs.CV cs.AI cs.MM 62%

Annotation-free Audio-Visual Segmentation

Jinxiang Liu, Yu Wang, Chen Ju, Chaofan Ma, Ya Zhang, Weidi Xie

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

Comments Camera-ready version for WACV 2024; project page is https://jinxiang-liu.github.io/anno-free-AVS/

2309.16569 2023-09-29 cs.SD cs.CV cs.MM eess.AS 62%

Audio-Visual Speaker Verification via Joint Cross-Attention

R. Gnana Praveen, Jahangir Alam

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

2309.06176 2023-09-13 cs.CV cs.MM 62%

Dual-Path Temporal Map Optimization for Make-up Temporal Video Grounding

Jiaxiu Li, Kun Li, Jia Li, Guoliang Chen, Dan Guo, Meng Wang

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；分类 cs.CV、cs.MM

2309.04734 2023-09-12 cs.CV cs.CL cs.MM 62%

Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering

Yifan Dong, Suhang Wu, Fandong Meng, Jie Zhou, Xiaoli Wang, Jianxin Lin, Jinsong Su

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；分类 cs.CV、cs.MM

Comments Accepted In Proceedings of the 31st ACM International Conference on Multimedia (MM' 23)

2307.10813 2023-07-21 cs.CV cs.SD eess.AS eess.IV 62%

Perceptual Quality Assessment of Omnidirectional Audio-visual Signals

Xilei Zhu, Huiyu Duan, Yuqin Cao, Yuxin Zhu, Yucheng Zhu, Jing Liu, Li Chen, Xiongkuo Min, Guangtao Zhai

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、eess.IV

Comments 12 pages, 5 figures, to be published in CICAI2023

2306.10567 2023-06-21 eess.AS cs.CV cs.MM cs.SD 62%

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

Yuchen Hu, Chen Chen, Ruizhe Li, Heqing Zou, Eng Siong Chng

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments 14 pages, 5 figures, Accepted by ACL 2023

2305.01836 2023-05-04 cs.CV cs.LG cs.MM cs.SD eess.AS 62%

AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation

Shentong Mo, Yapeng Tian

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

2207.04213 2023-03-07 cs.MM cs.CV cs.LG cs.SD eess.AS 62%

Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extraction

Zhongweiyang Xu, Xulin Fan, Mark Hasegawa-Johnson

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

Comments Paper Accepted by ICASSP2023

2212.01040 2022-12-05 cs.CV cs.MM cs.SD eess.AS 62%

Role of Audio in Audio-Visual Video Summarization

Ibrahim Shoer, Berkay Kopru, Engin Erzin

专题命中音视频/视觉语言融合：information fusion(abstract)；分类 cs.CV、cs.MM

2104.08510 2021-04-26 cs.MM cs.CV cs.SD eess.AS 62%

Exploring Deep Learning for Joint Audio-Visual Lip Biometrics

Meng Liu, Longbiao Wang, Kong Aik Lee, Hanyi Zhang, Chang Zeng, Jianwu Dang

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、cs.MM

2009.04107 2020-09-10 eess.AS cs.MM cs.SD eess.IV 62%

Multi-modal Attention for Speech Emotion Recognition

Zexu Pan, Zhaojie Luo, Jichen Yang, Haizhou Li

专题命中音视频/视觉语言融合：hybrid fusion(abstract)；分类 eess.IV、cs.MM

Comments Accepted by Interspeech2020

2004.09476 2020-04-21 cs.CV cs.LG cs.MM cs.SD eess.AS 62%

Music Gesture for Visual Sound Separation

Chuang Gan, Deng Huang, Hang Zhao, Joshua B. Tenenbaum, Antonio Torralba

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

Comments CVPR 2020. Project page: http://music-gesture.csail.mit.edu

1809.10961 2019-10-30 cs.CV cs.MM stat.ML 62%

Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers

Yutong Ban, Xavier Alameda-Pineda, Laurent Girin, Radu Horaud

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

1906.02497 2019-07-30 cs.IR cs.CV cs.MM 62%

Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos

Zhu Zhang, Zhijie Lin, Zhou Zhao, Zhenxin Xiao

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；分类 cs.CV、cs.MM

Comments Accepted by SIGIR 2019 as a full paper

Journal ref SIGIR, 2019, pages 655-664

1809.07999 2018-09-24 cs.CV cs.AI cs.CL cs.MM 62%

Multimodal Dual Attention Memory for Video Story Question Answering

Kyung-Min Kim, Seong-Ho Choi, Jin-Hwa Kim, Byoung-Tak Zhang

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments Accepted for ECCV 2018

2508.16300 2025-08-25 cs.CV cs.AI 61%

A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehension

Mohammad Zia Ur Rehman, Devraj Raghuvanshi, Umang Jain, Shubhi Bansal, Nagendra Kumar

机构 * Indian Institute of Technology Indore（印度印度理工学院Indore）； Brown University（布朗大学）

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV；information fusion(comments)

Comments Published in Information Fusion

2608.23102 2026-08-25 cs.CV cs.IR 新提交 57%

Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models

无需训练的伪融合：基于扩散模型和多模态大语言模型的组合图像检索

Fan Xu, Luis A. Leiva

机构 * University of Luxembourg（卢森堡大学）

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV

AI总结本文提出无需训练的PeFuse伪融合框架，结合扩散模型与多模态大语言模型，将组合图像检索转化为单模态检索任务，在零样本场景下实现了媲美当前最优方法的性能。

Journal ref Transactions on Machine Learning Research, 2026

2606.11269 2026-08-25 cs.CV cs.HC 版本更新 57%

Traits Run Deeper: Trait-Specific Asymmetric Fusion for Personality Assessment

特质更深：面向人格评估的特质特异性非对称融合

Jia Li, Qian Chen, Wei Wang, Xinyu Li, Zhenzhen Hu, Dongsheng Shao, Richang Hong, Meng Wang

机构 * Hefei University of Technology（合肥工业大学）； Intelligent Interconnected Systems Laboratory of Anhui Province（安徽省智能互联系统实验室）； Jianghuai Advanced Technology Center（江淮前沿技术中心）； Anhui Provincial Industry Innovation Center of Humanoid Robots（安徽省人形机器人产业创新中心）； Anhui Provincial Key Laboratory of Humanoid Robots（安徽省人形机器人重点实验室）

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV

AI总结提出Traits Run Deeper框架，通过多模态基础表示、特质特异性非对称融合和分布校准回归模块，解决人格评估中模态偏好差异和标签偏差问题，在AVI Challenge 2026上MSE降低约25%。

2511.17681 2026-08-25 cs.CV 版本更新 57%

Motion-Aware Vision-Reference Alignment for Referring Multi-Object Tracking

基于多模态大语言模型的视觉-运动-参考对齐的指称多目标跟踪

Weiyi Lv, Ning Zhang, Hanyang Sun, Haoran Jiang, Kai Zhao, Yixiao Gu, Jing Xiao, Dan Zeng

机构 * Shanghai University（上海大学）； PAII Inc.（PAII公司）

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；分类 cs.CV

AI总结 VMRMOT通过多模态大语言模型实现视觉-运动-参考对齐，提升指称多目标跟踪的性能。

Comments Accepted by IEEE Transactions on Multimedia

2608.19710 2026-08-21 cs.CV cs.AI 新提交 57%

Robust Cross-Modal Foundation Model Perception for Underwater Robots under Degraded Visual Conditions

退化视觉条件下水下机器人的鲁棒跨模态基础模型感知

Mohammad Arif Ul Alam

机构 * College of Science and Technology, North Carolina A & T State University（北卡罗来纳农工州立大学科学技术学院）

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV

AI总结该研究针对水下机器人视觉退化问题，提出感知退化的视觉-声纳门控融合方法，在极端退化下使平衡准确率较 DINOv2 基线提升 33.5%，实现鲁棒跨模态感知。

2607.15779 2026-08-12 cs.CV 版本更新 57%

Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion

通过交叉注意力和门控融合进行多模态矛盾与犹豫识别

Oussama Berhili, Yassine Ouzar, Larbi Boubchir

机构 * University of Paris 8（巴黎第八大学）； LIASD Laboratory（LIASD实验室）

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV

AI总结为ECCV 2026的ABAW11挑战赛开发多模态框架，用预训练编码器提取多模态特征，建立单模态基线，在此基础上提出含交叉注意力和门控融合的多模态架构，验证集宏F1达0.7394，提升显著。

2604.23957 2026-08-11 cs.CV 版本更新 57%

LAVA: Layered Audio-Visual Anti-tampering Watermarking for Robust Deepfake Detection and Localization

LAVA：分层音频视觉抗篡改水印用于鲁棒深度伪造检测与定位

Bokang Zeng, Zheng Gao, Xiaoyu Li, Xiaoyan Feng, Jiaojiao Jiang

机构 * School of Computer Science and Engineering（计算机科学与工程学院）； UNSW Sydney（新南威尔士大学悉尼分校）； School of Information and Communication Technology（信息与通信技术学院）； Griffith University（格里菲斯大学）

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV

AI总结 LAVA通过跨模态水印融合与校准感知对齐，提升深度伪造篡改检测与定位的鲁棒性，实验表明其检测性能接近完美，抗压缩与多模态错位能力强。

Comments Accepted at the 34th ACM International Conference on Multimedia (ACM MM 2026)

2509.16609 2026-08-11 cs.CV 版本更新 57%

Describe-to-Score: A text-guided framework for image complexity assessment

描述到评分：一种用于图像复杂度评估的文本引导框架

Shipeng Liu, Liang Zhao, Dengfeng Chen, Zhonglin Zhang

机构 * Xi’an University of Architecture and Technology（西安建筑科技大学）

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；分类 cs.CV

AI总结该研究针对现有图像复杂度评估方法无法捕捉高级语义的问题，提出文本引导框架D2S，仅训练阶段引入描述语义正则化，在IC9600基准达SOTA，跨任务也具竞争力。

Comments Accepted by Pattern Recognition

2608.06943 2026-08-10 cs.CV 新提交 57%

Dual-Space Modality Consistency Learning for Universal Cross-Modal Re-Identification

用于通用跨模态行人重识别的双空间模态一致性学习

Yujian Zhao, Yukang Zhao, Hankun Liu, Haoxuan Xu, Bo Li, Hanzi Wan, Guanglin Niu

机构 * School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）； School of Humanities and Social Sciences, Beihang University（北京航空航天大学人文与社会科学高等研究院）； School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）； School of Informatics, Xiamen University（厦门大学信息学院）

专题命中音视频/视觉语言融合：visible-infrared(abstract)；分类 cs.CV

AI总结提出DSMCL即插即用框架，联合建模空间与频域一致性，在5个数据集的17项协议上提升跨模态ReID基线性能，适配多样化异构模态场景。

2608.03475 2026-08-05 cs.MM cs.AI cs.CL 新提交 57%

Adaptive Modality Reliability Diagnosis and Restoration for Robust Multimodal Intent Recognition

用于鲁棒多模态意图识别的自适应模态可靠性诊断与恢复

Suraj Kumar, Mohnish Raj, Soumi Chattopadhayay, Chandranath Adak, Ayan Dutta

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.MM

AI总结提出PRIME闭环可靠性引导框架，通过诊断恢复多模态意图识别的不可靠模态，在保持干净数据性能的同时提升了多模态数据各类退化场景下的鲁棒性。

1. 音视频/视觉语言融合 498 篇

Straight Through Gumbel Softmax Estimator based Bimodal Neural Architecture Search for Audio-Visual Deepfake Detection

Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding

PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction

Efficient Selective Audio Masked Multimodal Bottleneck Transformer for Audio-Video Classification

VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment

Annotation-free Audio-Visual Segmentation

Audio-Visual Speaker Verification via Joint Cross-Attention

Dual-Path Temporal Map Optimization for Make-up Temporal Video Grounding

Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering

Perceptual Quality Assessment of Omnidirectional Audio-visual Signals

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation

Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extraction

Role of Audio in Audio-Visual Video Summarization

Exploring Deep Learning for Joint Audio-Visual Lip Biometrics

Multi-modal Attention for Speech Emotion Recognition

Music Gesture for Visual Sound Separation

Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers

Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos

Multimodal Dual Attention Memory for Video Story Question Answering

A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehension

Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models

Traits Run Deeper: Trait-Specific Asymmetric Fusion for Personality Assessment

Motion-Aware Vision-Reference Alignment for Referring Multi-Object Tracking

Robust Cross-Modal Foundation Model Perception for Underwater Robots under Degraded Visual Conditions

Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion

LAVA: Layered Audio-Visual Anti-tampering Watermarking for Robust Deepfake Detection and Localization

Describe-to-Score: A text-guided framework for image complexity assessment

Dual-Space Modality Consistency Learning for Universal Cross-Modal Re-Identification

Adaptive Modality Reliability Diagnosis and Restoration for Robust Multimodal Intent Recognition