多模态大模型 - arXivDaily 专题

1709.05038 2017-09-18 cs.CV cs.CL cs.LG 81%

Self-Guiding Multimodal LSTM - when we do not have a perfect training dataset for image captioning

Yang Xian, Yingli Tian

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments The paper is under consideration at Computer Vision and Image Understanding

1707.07601 2017-07-25 cs.CL cs.CV 81%

Image Pivoting for Learning Multilingual Multimodal Representations

Spandana Gella, Rico Sennrich, Frank Keller, Mirella Lapata

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments 7 pages, EMNLP 2017

1412.6632 2015-06-12 cs.CV cs.CL cs.LG 81%

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, Alan Yuille

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Add a simple strategy to boost the performance of image captioning task significantly. More details are shown in Section 8 of the paper. The code and related data are available at https://github.com/mjhucla/mRNN-CR ;. arXiv admin note: substantial text overlap with arXiv:1410.1090

Journal ref ICLR 2015

2608.01664 2026-08-04 cs.CV cs.LG 新提交 80%

FAU at ImageCLEF 2026 Task on Multimodal Reasoning Robust Candidate Scoring and Concise Multilingual Visual Answering

FAU参加2026年ImageCLEF多模态推理任务：鲁棒候选评分与简洁多语种视觉问答

Mohamed Basem, Vincent Christlein

机构 * Friedrich-Alexander-Universität Erlangen-Nürnberg（弗里德里希-亚历山大-埃尔兰根-纽伦堡大学）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV

AI总结 FAU提交的多模态推理系统，未做任务特定模型训练，在2026年ImageCLEF竞赛中，获Visual MCQ第三名、Visual OpenQA第一名，凸显推理工程的实用价值。

Comments 16 pages, 3 figures, 7 tables. CLEF 2026 Working Notes, ImageCLEF 2026 Multimodal Reasoning Task

2406.17582 2025-11-13 cs.HC cs.GR cs.MM 80%

Crafting Dynamic Virtual Activities with Advanced Multimodal Models

Changyang Li, Qingan Yan, Minyoung Kim, Zhan Li, Yi Xu, Lap-Fai Yu

专题命中图文多模态：multimodal(title,abstract)；分类 cs.MM

Journal ref C. Li, Q. Yan, M. Kim, Z. Li, Y. Xu and L. -F. Yu, "Crafting Dynamic Virtual Activities with Advanced Multimodal Models," 2025 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 120-130

2507.19196 2025-08-19 cs.RO cs.CL cs.HC 80%

Towards Multimodal Social Conversations with Robots: Using Vision-Language Models

Ruben Janssens, Tony Belpaeme

机构 * Ghent University–imec（根特大学–imec）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CL

Comments Accepted at the workshop "Human - Foundation Models Interaction: A Focus On Multimodal Information" (FoMo-HRI) at IEEE RO-MAN 2025 (Camera-ready version)

2504.09914 2025-04-15 cs.CV 80%

Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledge

Maria Tzelepi, Vasileios Mezaris

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV

Comments Accepted for publication, Multimodal Learning and Applications Workshop (MULA 2025) @ IEEE/CVF CVPR 2025, Nashville, TN, USA, June 2025. This is the authors' "accepted version"

2309.03452 2024-08-05 cs.CV cs.LG 80%

Multimodal Guidance Network for Missing-Modality Inference in Content Moderation

Zhuokai Zhao, Harish Palani, Tianyi Liu, Lena Evans, Ruth Toner

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV

Comments ICME 2024 Camera Ready. Code is available at https://github.com/zhuokaizhao/multimodal-guidance-network

2603.22287 2026-03-25 cs.CV cs.AI cs.CL 80%

Founder effects shape the evolutionary dynamics of multimodality in open LLM families

奠基效应塑造了开放大语言模型家族中多模态的进化动态

Manuel Cebrian

机构 * Center for Automation and Robotics, Spanish National Research Council, Madrid, Spain（自动化与机器人中心，西班牙国家研究委员会，西班牙马德里）

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结研究通过分析模型谱系数据，发现多模态能力在开放大语言模型家族中通过罕见的奠基事件进入，并在后代谱系中快速扩展，导致多模态能力的采用动态呈现突变特征。

Comments 7 pages, 4 figures, 2 tables

2510.11852 2025-10-15 cs.LG 80%

Evaluating Open-Source Vision-Language Models for Multimodal Sarcasm Detection

Saroj Basnet, Shafkat Farabi, Tharindu Ranasinghe, Diptesh Kanoji, Marcos Zampieri

机构 * George Mason University（乔治·马歇尔大学）； Lancaster University（兰卡斯特大学）； University of Surrey（萨里大学）

专题命中图文多模态：multimodal(title,abstract)

Comments Accepted to ICDMW 2025 Workshop on Multimodal AI (MMAI). Full workshop info: https://icdmw25mmai.github.io/

Journal ref Proc. IEEE International Conference on Data Mining Workshops (ICDMW 2025), Workshop on Multimodal AI (MMAI 2025), Los Angeles, USA, December 2025

2504.11695 2025-08-25 cs.CV cs.CL cs.MM 80%

Interpreting the linear structure of vision-language model embedding spaces

Isabel Papadimitriou, Huangyuan Su, Thomas Fel, Sham Kakade, Stephanie Gil

机构 * Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University（哈佛大学自然与人工智能研究所）； Department of Computer Science, Harvard University（哈佛大学计算机科学系）

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments COLM 2025

2502.15392 2025-02-24 cs.AI cs.CL cs.CV 80%

Chitrarth: Bridging Vision and Language for a Billion People

Shaharukh Khan, Ayush Tarun, Abhinav Ravi, Ali Faraz, Akshat Patidar, Praveen Kumar Pokala, Anagha Bhangare, Raja Kolla, Chandra Khatri, Shubham Agarwal

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.CL、cs.AI

2305.01278 2023-10-25 cs.CV cs.AI cs.CL 80%

VPGTrans: Transfer Visual Prompt Generator across LLMs

Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, Tat-Seng Chua

专题命中图文多模态：multimodal(abstract)；MLLM(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Website: https://vpgtrans.github.io Code: https://github.com/VPGTrans/VPGTrans NeurIPS 2023

2211.13437 2023-03-28 cs.CV cs.CL cs.MM 80%

Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learning

Yatai Ji, Rongcheng Tu, Jie Jiang, Weijie Kong, Chengfei Cai, Wenzhe Zhao, Hongfa Wang, Yujiu Yang, Wei Liu

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments CVPR 2023 accept

2211.12737 2022-11-24 cs.CV cs.AI cs.CL cs.LG 80%

RoentGen: Vision-Language Foundation Model for Chest X-ray Generation

Pierre Chambon, Christian Bluethgen, Jean-Benoit Delbrouck, Rogier Van der Sluijs, Małgorzata Połacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P. Langlotz, Akshay Chaudhari

专题命中图文多模态：multimodal(abstract)；multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 19 pages

2608.15056 2026-08-18 cs.AI 新提交 79%

GraphLoom: Reliability-Calibrated Graph Evidence Routing for Multimodal KG-RAG

GraphLoom：面向多模态KG-RAG的可靠性校准图证据路由

Zafar Ali, Asad Khan, Aalia Malik, Pavlos Kefalas

机构 * School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）； Aristotle University（亚里士多德大学）； Dashub（达舒布）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.AI

AI总结提出GraphLoom框架，通过可靠性校准的图证据路由实现多模态KG-RAG，在ScienceQA等数据集上提升了答案质量与证据忠实度。

2503.06223 2026-08-18 cs.CV 79%

RedDiffuser: Auditing Multimodal Safety Failures in Vision-Language Models via Reinforced Diffusion

RedDiffuser：通过强化扩散审计多模态安全故障

Ruofan Wang, Xingjun Ma

机构 * Fudan University（复旦大学）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV

AI总结研究多模态系统在有害上下文暴露下的安全审计，提出RedDiffuser框架通过扩散模型生成视觉输入，揭示隐藏的安全漏洞，实验显示VLMs在部分有毒文本与视觉上下文结合时存在广泛安全问题。

2608.13861 2026-08-17 cs.CV 新提交 79%

XSA-MAD: Cross-modal Semantic Alignment for Morphing Attack Detection

XSA-MAD：用于变形攻击检测的跨模态语义对齐

Jie Jin, Mahiro Tokumasu, Yu Makino, Masakatsu Nishigaki, Tetsushi Ohki

机构 * RIKEN AIP（理化学研究所人工智能项目）

专题命中图文多模态：cross-modal(title)；multimodal(abstract)；分类 cs.CV

AI总结针对现有图像型变形攻击检测方法泛化性差的问题，提出基于CLIP的多模态框架XSA-MAD，通过对齐图像与语义空间实现对不同变形攻击的检测，在高保真攻击下性能优于现有方法

Comments accepted to ICIP2026

2608.11013 2026-08-12 cs.CV 新提交 79%

Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning

观看合成视频：针对零样本视频字幕生成的视觉合成跨模态表征对齐

Liangyu Fu, Junbo Wang, Yuke Li, Ya Jing, Xuecheng Wu, Zhiyong Wang

机构 * School of Software, Northwestern Polytechnical University（西北工业大学软件学院）； School of Information Science and Technology, Beijing University of Technology（北京工业大学信息科学与技术学院）； School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）； School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）

专题命中图文多模态：cross-modal(title,abstract)；分类 cs.CV

AI总结本文提出WSV零样本视频字幕生成框架，通过文本到视频生成模型、抛光器、提示器弥合跨模态差距，在三个公开数据集上取得了B@4 52、CIDEr 95.7的成绩。

2608.10408 2026-08-12 cs.CL 新提交 79%

VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback?

VisEditBench：视觉语言模型能否基于多模态反馈编辑可视化代码？

Mizanur Rahman, Arshia Azimlu, Shadikur Rahman, Md Tahmid Rahman Laskar, Amran Bhuiyan, Shafiq Joty, Enamul Hoque Prince

机构 * York University（约克大学）； Nanyang Technological University（南洋理工大学）； Salesforce AI Research（Salesforce人工智能研究）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CL

AI总结本研究推出可视化代码编辑基准VisEditBench，评估20种VLMs的编辑能力，提出基于渲染的VisEditAgent框架，显著提升了编辑任务的通过率。

2608.09202 2026-08-11 cs.AI 新提交 79%

CRUISE: Vision-Language Model-Guided Uncertainty-Aware Cross-Modal Sensor Fusion for Robust Autonomous Driving

CRUISE：面向鲁棒自动驾驶的视觉语言模型引导的不确定性感知跨模态传感器融合

Junyao Wang, Yulin Xu, Yu Li, Pramod Khargonekar, Mohammad Abdullah Al Faruque

机构 * University of California, Irvine（加利福尼亚大学欧文分校）

专题命中图文多模态：cross-modal(title,abstract)；分类 cs.AI

AI总结针对现有不确定性感知融合方法泛化性差的问题，提出CRUISE框架，结合VLM引导的UQ模块与动态自适应机制，实现鲁棒的自动驾驶跨模态传感器融合。

2608.05833 2026-08-11 cs.AI 版本更新 79%

ViSR-KGC: Visual Subgraph Reasoning with Vision-Language Models for Multimodal Knowledge Graph Completion

ViSR-KGC：结合视觉语言模型的视觉子图推理用于多模态知识图谱补全

Jiafan Li, Mengxue Yang, Jiaqi Zhu, Liang Chang, Ying Li, Hongan Wang

机构 * Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）； University of Chinese Academy of Sciences（中国科学院大学）； CITIC Securities（中信证券）； School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.AI

AI总结针对多模态知识图谱补全的痛点，提出ViSR-KGC方法，结合表示学习、视觉语言模型与预训练常识，通过提取查询感知子图并转换为可视化内容辅助VLM推理缺失实体。

2604.11402 2026-08-07 cs.CV 版本更新 79%

SCD4VPR: Multi-modal Scene Change Detection for Long-term Visual Place Recognition Database Update

基于视觉-语言表示学习的场景变化检测

Diwei Sheng, Vijayraj Gohil, Satyam Gaba, Zihan Liu, Giles Hamilton-Fletcher, John-Ross Rizzo, Yongqing Liang, Chen Feng

机构 * New York University（纽约大学）

专题命中图文多模态：multi-modal(title)；cross-modal(abstract)；分类 cs.CV

AI总结本文提出LangSCD框架，通过融合语言和视觉信息提升场景变化检测的鲁棒性，引入几何-语义匹配模块和多类标注数据集NYC-CD，实验证明其在多个基准上的优越性能。

Comments 9 pages, 6 figures, 6 tables

2608.03322 2026-08-05 cs.CV 新提交 79%

LocAnyMed: Vision-Language Grounding for Multimodal Medical Images

LocAnyMed：面向多模态医学图像的视觉-语言定位

Zihan Wang, Tong Liu, Zhiwei Wang, Tao Huang, Wentao Jiang, Sihan Ma, Shanshan Ye, Xiaohui Yang, Jing Zhang

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV

AI总结该研究构建多模态医学视觉定位数据集LocAnyMed-200K及理由增强子集LocAnyMed-CoT-20K，通过微调LocateAnything-3B提升医学定位性能，为相关研究提供统一基础。

Comments Technical report; work in progress. 28 pages, 5 figures, and 16 tables. Code: https://github.com/MiliLab/LocAnyMed

2607.21552 2026-07-24 cs.AI cs.LG 新提交 79%

MIRROR: Learning from the Other View for Multi-Modal Reasoning

MIRROR：从其他视角学习以进行多模态推理

Wen Ye, Yuxiao Qu, Aviral Kumar, Xuezhe Ma

机构 * University of Southern California（南加州大学）； Carnegie Mellon University（卡内基梅隆大学）

专题命中图文多模态：multi-modal(title)；multimodal(abstract)；分类 cs.AI

AI总结研究视觉语言模型在几何问题多模态推理上的不足，构建ODA-Data数据集，提出模态感知互惠推理优化（MIRROR）方法，通过强化学习自我监督，提升多模态推理能力，在相关基准测试中表现更优。

2607.20284 2026-07-23 cs.CV 新提交 79%

Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?

用于遥感图像理解的多模态大语言模型：领域特定还是通用？

Qiwei Ma, Chunping Qiu, Xinjun Cheng, Xiaoyu Zhang, Puhong Duan, Ke Yang, Xudong Kang, Shutao Li

机构 * School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）； Intelligent Game and Decision Lab (IGDL)（智能游戏与决策实验室）； Yuelushan Center for Industrial Innovation（岳麓山工业创新中心）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV

AI总结本文针对遥感图像理解的多模态大语言模型展开研究，通过系统调查和评估，比较其与通用模型在不同任务上的表现，发现当前模型存在局限，进而给出未来方向，为开发相关模型提供系统参考。

Comments 27 pages, 11 figures

2607.18615 2026-07-22 cs.CL cs.LG 新提交 79%

Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning

随机元遗忘：连接语言主干与多模态遗忘

Zijie Liu, Jinhao Duan, Gaowen Liu, Sijia Liu, Tianlong Chen

机构 * UNC at Chapel Hill（北卡罗来纳大学教堂山分校）； Cisco Research（思科研究院）； Michigan State University（密歇根州立大学）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CL

AI总结研究视觉语言模型的机器遗忘问题，提出随机元遗忘框架，利用VLM级反馈学习初始化，经内、外循环更新语言主干，实验证明该方法在遗忘-保留权衡上表现最佳，能提升准确率且可迁移。

Comments 15 pages

2509.25699 2026-07-22 cs.CV 版本更新 79%

AIM-CoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning

AIM-CoT：基于主动信息的多模态链式推理用于视觉-语言推理

Xiping Li, Jianghong Ma

机构 * The Chinese University of Hong Kong（香港中文大学）； Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））

专题命中图文多模态：multimodal(title)；multi-modal(abstract)；分类 cs.CV

AI总结本文提出AIM-CoT框架，通过上下文增强注意力图生成、主动视觉探测和动态注意力转移触发，改进视觉语言模型的证据选择与插入触发，提升视觉-语言推理性能。

Comments Accepted by ACL 2026 Main Conference. 30 pages, 6 figures

2607.16243 2026-07-21 cs.LG cs.AI 新提交 79%

RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants

RobustMAD：评估用于可部署异常检测助手的多模态小语言模型的现实世界鲁棒性

Anushiya Arunan, Xin Li, Yan Qin, U-Xuan Tan, Nhu Khue Vuong, Xiaoli Li, Chau Yuen

机构 * Singapore University of Technology and Design（新加坡科技设计大学）； Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）； Nanyang Technological University（南洋理工大学）； Chongqing University（重庆大学）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.AI

AI总结研究针对多模态小语言模型现场部署的鲁棒性评估问题，开发RobustMAD基准测试，通过多种开放式查询评估模型，发现最佳模型虽有潜力但仍存差距及三种失败模式，为下一代多模态工业检测助手设计提供指导。

Comments Accepted for publication in Transactions on Machine Learning Research (TMLR), 2026 at https://openreview.net/forum?id=skrA9UYNIZ

2607.11436 2026-07-20 cs.AI 版本更新 79%

The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoning

多模态焦点的潮起潮落：为基于视觉的语言模型推理调度视觉中继窗口

Wencheng Ye, Yi Bin, Yujuan Ding, Hongye Fang, Zheng Wang, Xing Xu, Jingkuan Song, Yun Zhang, Sirui Da, Heng Tao Shen

机构 * School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）； School of Fashion and Textiles, The Hong Kong Polytechnic University（香港理工大学纺织及制衣学院）； School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.AI

AI总结研究视觉语言模型视觉证据不稳定问题，通过剖析其内部多模态注意力焦点的三阶段分布，提出TRACE框架，该框架能自适应控制推理，在多模型和多基准测试中显著提升基于证据的多模态推理能力。

1. 图文多模态 4657 篇

Self-Guiding Multimodal LSTM - when we do not have a perfect training dataset for image captioning

Image Pivoting for Learning Multilingual Multimodal Representations

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

FAU at ImageCLEF 2026 Task on Multimodal Reasoning Robust Candidate Scoring and Concise Multilingual Visual Answering

Crafting Dynamic Virtual Activities with Advanced Multimodal Models

Towards Multimodal Social Conversations with Robots: Using Vision-Language Models

Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledge

Multimodal Guidance Network for Missing-Modality Inference in Content Moderation

Founder effects shape the evolutionary dynamics of multimodality in open LLM families

Evaluating Open-Source Vision-Language Models for Multimodal Sarcasm Detection

Interpreting the linear structure of vision-language model embedding spaces

Chitrarth: Bridging Vision and Language for a Billion People

VPGTrans: Transfer Visual Prompt Generator across LLMs

Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learning

RoentGen: Vision-Language Foundation Model for Chest X-ray Generation

GraphLoom: Reliability-Calibrated Graph Evidence Routing for Multimodal KG-RAG

RedDiffuser: Auditing Multimodal Safety Failures in Vision-Language Models via Reinforced Diffusion

XSA-MAD: Cross-modal Semantic Alignment for Morphing Attack Detection

Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning

VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback?

CRUISE: Vision-Language Model-Guided Uncertainty-Aware Cross-Modal Sensor Fusion for Robust Autonomous Driving

ViSR-KGC: Visual Subgraph Reasoning with Vision-Language Models for Multimodal Knowledge Graph Completion

SCD4VPR: Multi-modal Scene Change Detection for Long-term Visual Place Recognition Database Update

LocAnyMed: Vision-Language Grounding for Multimodal Medical Images

MIRROR: Learning from the Other View for Multi-Modal Reasoning

Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?

Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning

AIM-CoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning

RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants

The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoning