视觉大模型 / VLM

Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models

多模态大语言模型中模态干扰的诊断与缓解

Rui Cai, Bangzheng Li, Xiaofei Wen, Muhao Chen, Zhe Zhao

机构 * Department of Computer Science, University of California-Davis, Davis, CA, United States（计算机科学系，加州大学戴维斯分校） 纠错

专题命中 视觉问答 ：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI、cs.LG

AI总结 本文提出了一种统一的微调框架，通过结合启发式和对抗性扰动的数据增强与输出一致性正则化，有效缓解多模态大语言模型中的模态干扰问题，提升模型的鲁棒性和泛化能力。

SurgPub-Video: A Comprehensive Surgical Video Dataset for Enhanced Surgical Intelligence in Vision-Language Model

SurgPub-Video: 一个全面的外科视频数据集，用于增强视觉-语言模型中的外科智能

Yaoqian Li, Xikai Yang, Dunyuan Xu, Yang Yu, Litao Zhao, Xiaowei Hu, Jinpeng Li, Pheng-Ann Heng

专题命中 视觉问答 ：vision-language model(title,abstract)；VLM(abstract)；visual question answering(abstract)

AI总结 SurgPub-Video数据集和SurgLLaVA-Video模型通过提供高质量外科视频数据和专门的视觉-语言模型，提升了手术场景分析的智能水平。

Journal ref AAAI-2026

MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering

Varun Srivastava, Fan Lei, Srija Mukhopadhyay, Vivek Gupta, Ross Maciejewski

机构 * School of Computing and Augmented Intelligence（计算与增强智能学院） ； Arizona State University（亚利桑那州立大学） ； Department of Computer Science（计算机科学系） ； International Institute of Information Technology（国际信息科技研究所） 纠错

专题命中 视觉问答 ：multimodal large language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Published as a conference paper at COLM 2025

Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models

Jinyeong Kim, Seil Kang, Jiwoo Park, Junhyeok Kim, Seong Jae Hwang

专题命中 视觉问答 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning

Songtao Jiang, Yuxi Chen, Sibo Song, Yan Zhang, Yeying Jin, Yang Feng, Jian Wu, Zuozhu Liu

机构 * Zhejiang University, Zhejiang, China（浙江大学） ； Alibaba Group, Zhejiang, China（阿里巴巴集团） ； Angelalign Technology Inc., Shanghai, China（Angelalign技术有限公司） ； ChohoTech Inc., Hangzhou, China（楚合科技有限公司） ； Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence, Zhejiang, China（浙江省医学影像人工智能重点实验室） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)；vision-language model(abstract)；LLaVA(abstract)

Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger

Qi Yang, Chenghao Zhang, Lubin Fan, Kun Ding, Jieping Ye, Shiming Xiang

机构 * School of Artificial Intelligence, University of Chinese Academy of Sciences, China（中国科学院大学人工智能学院） ； MAIS, Institute of Automation, Chinese Academy of Sciences, China（中国科学院自动化研究所MAIS部） ； Alibaba Cloud Computing, China（阿里巴巴云计算） 纠错

专题命中 视觉问答 ：vision-language model(title)；vision language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments ICML 2025 Spotlight. 22 pages, 16 figures

ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understanding

Ankit Pal, Jung-Oh Lee, Xiaoman Zhang, Malaikannan Sankarasubbu, Seunghyeon Roh, Won Jung Kim, Meesun Lee, Pranav Rajpurkar

机构 * Saama AI Research（Saama AI研究机构） ； Seoul National University（首尔国立大学） ； Harvard Medical School（哈佛医学院） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Value-Spectrum: Quantifying Preferences of Vision-Language Models via Value Decomposition in Social Media Contexts

Jingxuan Li, Yuning Yang, Shengqi Yang, Linfan Zhang, Ying Nian Wu

机构 * University of California, Los Angeles（加州大学洛杉矶分校） ； Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室） 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)；VLM(abstract)；visual question answering(abstract)

Comments ACL 2025 main

Fast or Slow? Integrating Fast Intuition and Deliberate Thinking for Enhancing Visual Question Answering

Songtao Jiang, Chenyi Zhou, Yan Zhang, Yeying Jin, Zuozhu Liu

机构 * Zhejiang University（浙江大学） ； Byte Dance（字节跳动） ； National University of Singapore（新加坡国立大学） ； Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)；multimodal large language model(abstract)；MLLM(abstract)

Spa-VLM: Stealthy Poisoning Attacks on RAG-based VLM

Lei Yu, Yechao Zhang, Ziqi Zhou, Yang Wu, Wei Wan, Minghui Li, Shengshan Hu, Pei Xiaobing, Jing Wang

专题命中 视觉问答 ：VLM(title,abstract)；vision-language model(abstract)；visual question answering(abstract)

Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Ugly

Lianghan Dong, Anamaria Crisan

专题命中 视觉问答 ：vision language model(title,abstract)；VLM(abstract)；LLaVA(abstract)

Advancing Egocentric Video Question Answering with Multimodal Large Language Models

Alkesh Patel, Vibhav Chitalia, Yinfei Yang

机构 * Apple（苹果公司） 纠错

专题命中 视觉问答 ：multimodal large language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 8 pages

UniRAG: Universal Retrieval Augmentation for Large Vision Language Models

Sahel Sharifymoghaddam, Shivani Upadhyay, Wenhu Chen, Jimmy Lin

专题命中 视觉问答 ：vision language model(title,abstract)；LLaVA(abstract)；visual question answering(abstract)

Comments 14 pages, 6 figures

Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types

Neelabh Sinha, Vinija Jain, Aman Chadha

机构 * Georgia Institute of Technology（佐治亚理工学院） ； Meta AI ； Amazon GenAI（亚马逊生成式人工智能部门） 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)；InternVL(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted at The First Workshop of Evaluation of Multi-Modal Generation (EvalMG) in 31st International Conference on Computational Linguistics (COLING), 2025. 8 pages + references + 6 pages of Appendix

Effectiveness Assessment of Recent Large Vision-Language Models

Yao Jiang, Xinyu Yan, Ge-Peng Ji, Keren Fu, Meijun Sun, Huan Xiong, Deng-Ping Fan, Fahad Shahbaz Khan

专题命中 视觉问答 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted by Visual Intelligence

Journal ref Visual Intelligence, 2024, Vol. 2, article no. 17

Vision-Language Model Based Handwriting Verification

Mihir Chauhan, Abhishek Satbhai, Mohammad Abuzar Hashemi, Mir Basheer Ali, Bina Ramamurthy, Mingchen Gao, Siwei Lyu, Sargur Srihari

专题命中 视觉问答 ：vision-language model(title)；vision language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 4 Pages, 1 Figure, 1 Table, Accepted as Short paper at Irish Machine Vision and Image Processing (IMVIP) Conference

Vision-Language Models as Success Detectors

Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, Serkan Cabi

专题命中 视觉问答 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Flamingo: a Visual Language Model for Few-Shot Learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, Karen Simonyan

专题命中 视觉问答 ：visual language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 54 pages. In Proceedings of Neural Information Processing Systems (NeurIPS) 2022

'Just because you are right, doesn't mean I am wrong': Overcoming a Bottleneck in the Development and Evaluation of Open-Ended Visual Question Answering (VQA) Tasks

Man Luo, Shailaja Keyur Sampat, Riley Tallman, Yankai Zeng, Manuha Vancha, Akarshan Sajja, Chitta Baral

专题命中 视觉问答 ：visual question answering(title,abstract)；vision-language model(abstract)；visual reasoning(abstract)

Comments accepted to EACL 2021

What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Learning

Jae Hee Lee, Matthias Kerzel, Kyra Ahrens, Cornelius Weber, Stefan Wermter

专题命中 视觉问答 ：grounding(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted to IJCAI 2022

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Drew A. Hudson, Christopher D. Manning

专题命中 视觉问答 ：visual reasoning(title,abstract)；grounding(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Published as a conference paper at CVPR 2019 (oral)

Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

Aishwarya Agrawal, Dhruv Batra, Devi Parikh, Aniruddha Kembhavi

专题命中 视觉问答 ：visual question answering(title,abstract)；grounding(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 15 pages, 10 figures. To appear in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions

Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, Aman Chadha

机构 * Department of Computer Science and Engineering, IIT Patna, India（印度帕纳大学计算机科学与工程系） ； Stanford University（斯坦福大学） ； Amazon AI（亚马逊人工智能） ； Indian Institute of Technology Patna, India（印度帕纳大学） 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI；visual language model(comments)

Comments One of the first survey on Visual Language Models

GLIMPSE: Holistic Cross-Modal Explainability for Large Vision-Language Models

Guanxi Shen

机构 * Georgia Institute of Technology（佐治亚理工学院） 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Keywords: Explainable Computer Vision, Large Vision-Language Models, AI Interpretability, Explainable AI, Visual Saliency, Attribution Maps, Cross-Modal Attribution, Human Attention Alignment, AI Transparency

Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimization

Kento Kawaharazuka, Yoshiki Obinata, Naoaki Kanazawa, Kei Okada, Masayuki Inaba

专题命中 视觉问答 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI；VLM(comments)

Comments Accepted at Advanced Robotics, website - https://haraduka.github.io/vlm-bbo/

Dynamic Clue Bottlenecks: Towards Interpretable-by-Design Visual Question Answering

Xingyu Fu, Ben Zhou, Sihao Chen, Mark Yatskar, Dan Roth

专题命中 视觉问答 ：visual question answering(title,abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments Multimodal, Visual Question Answering, Vision and Language

LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models

Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, Ping Luo

专题命中 视觉问答 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments 28 pages, 10 figures, a comprehensive evaluation of large vision-language models

MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agents

MetaSpace：面向具身智能体空间认知的变形测试

Gengyang Xu, Dongwei Xiao, Yiteng Peng, Shuai Wang

机构 * Hong Kong University of Science and Technology（香港科技大学） 纠错

专题命中 视觉问答 ：MLLM(summary_cn,abstract)；visual question answering(abstract)；分类 cs.AI

AI总结 MetaSpace是评估具身智能体空间认知的框架，基于变形测试原则生成测试用例，在三个场景中检测到SOTA MLLM驱动智能体的90422个空间认知错误，其SC分数远低于人类基准。

Comments 30 pages, 17 figures. Published in Proceedings of the ACM on Programming Languages (OOPSLA1)

Journal ref Proceedings of the ACM on Programming Languages, 10, OOPSLA1 (April 2026), 343-372

Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization

提示优化器是盲目运行的吗？用于自动提示优化的跨模态视觉反馈

Haoyue Liu, Xiaoyu Ma, Ye Chen, Yuexian Zou, Xiaoying Tang

专题命中 视觉问答 ：VLM(summary_cn,abstract)；vision-language model(abstract)；分类 cs.AI

AI总结 研究多模态任务中自动提示优化的盲目性问题，提出跨模态视觉反馈方法，含故障条件视觉诊断和错误感知聚合阶段，在多个VQA数据集和目标VLM上效果显著，能提升分数且优化器可跨模型转移。

Deterministic Hallucination Detection in Medical VQA via Confidence-Evidence Bayesian Gain

医学VQA中通过置信度-证据贝叶斯增益实现确定性幻觉检测

Mohammad Asadi, Tahoura Nedaee, Jack W. O'Sullivan, Euan Ashley, Ehsan Adeli

机构 * Department of Electrical Engineering, Stanford University, CA, USA（电气工程系，斯坦福大学） ； Department of Biology, Stanford University, CA, USA（生物学系，斯坦福大学） ； Division of Cardiology, Department of Medicine, Stanford University, CA, USA（心脏病学部，医学系，斯坦福大学） ； Department of Biomedical Data Science, Stanford University, CA, USA（生物医学数据科学系，斯坦福大学） ； Department of Computer Science, Stanford University, CA, USA（计算机科学系，斯坦福大学） ； Department of Psychiatry and Behavioral Sciences, Stanford University, CA, USA（精神病学与行为科学系，斯坦福大学） 纠错

专题命中 视觉问答 ：MLLM(summary_cn,abstract_cn)；visual question answering(abstract)；multimodal large language model(abstract)；分类 cs.AI

AI总结 本文提出CEBaG方法，利用模型自身log概率中的不一致置信度和弱视觉证据敏感性，实现无需随机采样和外部模型的确定性幻觉检测，在医疗MLLM和VQA基准测试中取得最佳AUC表现。