视觉大模型 / VLM

Efficient Multimodal Large Language Models: A Survey

高效多模态大语言模型：综述

Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, Yabiao Wang, Chengjie Wang, Lizhuang Ma

机构 * Youtu Lab, Tencent（腾讯优图实验室） ； SJTU（上海交通大学） ； BAAI（北京人工智能研究院） ； ECNU（华东师范大学） 纠错

专题命中 视觉问答 ：multimodal large language model(title,abstract)；visual question answering(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

AI总结 本文综述了高效多模态大语言模型的发展现状，探讨了其高效结构、策略及应用，并展望了未来研究方向。

Comments Accepted by Visual Intelligence

Journal ref Visual Intelligence, Volume 3, article number 27, (2025)

DermaBench: A Clinician-Annotated Benchmark Dataset for Dermatology Visual Question Answering and Reasoning

DermaBench：一种用于皮肤科视觉问答和推理的临床标注基准数据集

Abdurrahim Yilmaz, Ozan Erdem, Ece Gokyayla, Ayda Acar, Burc Bugra Dagtas, Dilara Ilhan Erdil, Gulsum Gencoglan, Burak Temelkuran

机构 * Imperial College London（帝国理工学院伦敦分校） ； Istanbul Medeniyet University（伊斯坦布尔医学大学） ； Usak Research and Training Hospital（Usak研究与培训医院） ； Istanbul Research and Training Hospital（伊斯坦布尔研究与培训医院） ； Ipswich Hospital（伊普斯韦奇医院） ； Medicana Atakoy Hospital（Medicana阿塔基医院） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)；vision-language model(abstract)；grounding(abstract)；分类 cs.CV、cs.AI

AI总结 DermaBench是首个由临床专家标注的皮肤科视觉问答基准数据集，通过精细标注和开放式描述提升多模态模型在皮肤科图像理解与临床推理中的评估能力。

Evaluating the Diagnostic Classification Ability of Multimodal Large Language Models: Insights from the Osteoarthritis Initiative

评估多模态大语言模型的诊断分类能力：来自骨关节炎倡议的见解

Li Wang, Xi Chen, XiangWen Deng, HuaHui Yi, ZeKun Jiang, Kang Li, Jian Li

专题命中 视觉问答 ：multimodal large language model(title,abstract)；visual question answering(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

AI总结 研究发现，多模态大语言模型在医学图像分类中表现不佳，建议优先优化视觉编码器和数据集质量。

Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perception

评估小型视觉-语言模型在距离依赖性交通感知中的表现

Nikos Theodoridis, Tim Brophy, Reenu Mohandas, Ganesh Sistu, Fiachra Collins, Anthony Scanlan, Ciaran Eising

机构 * University of Limerick（利默里克大学） ； Data Driven Computer Engineering Research Centre（数据驱动计算机工程研究中心） ； Lero, The Irish Software Research Centre（Lero爱尔兰软件研究中心） ； Valeo Vision Systems（瓦莱欧视觉系统） 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)；VLM(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

AI总结 本文评估了小型视觉-语言模型在距离依赖性交通感知任务中的表现，发现其在感知能力上显著弱于人类。

Comments Published in IEEE Open Journal of Vehicular Technology. Final version available at: https://ieeexplore.ieee.org/document/11230063

Journal ref IEEE Open Journal of Vehicular Technology, vol. 7, pp. 54-72, 2026

ZeShot-VQA: Zero-Shot Visual Question Answering Framework with Answer Mapping for Natural Disaster Damage Assessment

ZeShot-VQA：一种基于零样本学习的视觉问答框架，用于自然灾害损害评估

Ehsan Karimi, Maryam Rahnemoonfar

专题命中 视觉问答 ：visual question answering(title,abstract)；vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.LG

AI总结 本文提出基于零样本学习的ZeShot-VQA框架，用于自然灾害损害评估，无需微调即可处理新数据集并生成未见过的答案。

Comments Accepted by the 2025 IEEE International Geoscience and Remote Sensing Symposium (IGARSS 2025)

HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models

Sushant Gautam, Michael A. Riegler, Pål Halvorsen

机构 * Simula Metropolitan Center for Digital Engineering (SimulaMet)（Simula数字工程研究中心） ； Oslo Metropolitan University (OsloMet)（奥斯陆 Metropolitan 大学） ； Simula Research Laboratory（Simula研究实验室） 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)；LLaVA(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

CLARIFY: A Specialist-Generalist Framework for Accurate and Lightweight Dermatological Visual Question Answering

Aranya Saha, Tanvir Ahmed Khan, Ismam Nur Swapnil, Mohammad Ariful Haque

机构 * Aranya Saha ∗ , Tanvir Ahmed Khan ∗ , Ismam Nur Swapnil ∗ , and Mohammad Ariful Haque ∗（无明确机构） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)；vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 8 figures, Prepared for submission to IEEE Transactions on Human-Machine Systems

A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model

Zhe Xu, Ziyi Liu, Junlin Hou, Jiabo Ma, Cheng Jin, Yihui Wang, Zhixuan Chen, Zhengyu Zhang, Fuxiang Huang, Zhengrui Guo, Fengtao Zhou, Yingxue Xu, Xi Wang, Ronald Cheong Kin Chan, Li Liang, Hao Chen

机构 * Hong Kong University of Science and Technology（香港科技大学） ； Southern Medical University（南方医科大学） ； Nanfang Hospital and School of Basic Medical Sciences（南方医院和基础医学科学学院） ； Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 视觉问答 ：multimodal large language model(title,abstract)；visual question answering(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets

Kaiyuan Chen, Shuangyu Xie, Zehan Ma, Pannag R Sanketi, Ken Goldberg

专题命中 视觉问答 ：visual question answering(title,abstract)；vision-language model(abstract)；VLM(abstract)；分类 cs.AI、cs.LG

Scaling Large Vision-Language Models for Enhanced Multimodal Comprehension In Biomedical Image Analysis

Robinson Umeike, Neil Getty, Fangfang Xia, Rick Stevens

专题命中 视觉问答 ：vision-language model(title)；vision language model(abstract)；LLaVA(abstract)；visual question answering(abstract)

Comments 4 Pages, 4 Figures, 1 Table

Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving

Ming Liu, Siyuan Liang, Koushik Howlader, Liwen Wang, Dacheng Tao, Wensheng Zhang

机构 * Iowa State University（爱荷华州立大学） ； National University of Singapore（新加坡国立大学） 纠错

专题命中 视觉问答 ：vision language model(title)；vision-language model(abstract)；VLM(abstract)；visual question answering(abstract)

Data Metabolism: An Efficient Data Design Schema For Vision Language Model

Jingyuan Zhang, Hongzhi Zhang, Zhou Haonan, Chenxi Sun, Xingguang ji, Jiakang Wang, Fanheng Kong, Yahui Liu, Qi Wang, Fuzheng Zhang

机构 * Kuaishou Technology（快手科技） 纠错

专题命中 视觉问答 ：vision language model(title)；VLM(abstract)；visual language model(abstract)；visual question answering(abstract)

Comments To be presented at ICLR 2025, First Workshop on Open Science for Foundation Models

Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering

Federico Cocchi, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

机构 * University of Modena and Reggio Emilia（摩德纳大学和雷焦艾米利亚大学） ； University of Pisa（比萨大学） ； IIT-CNR（意大利国家研究委员会信息科学与技术研究所） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)；LLaVA(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments CVPR 2025

KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Language

Yoonshik Kim, Jaeyoon Jung

机构 * MAUM AI Inc.（MAUM AI公司） 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)；VLM(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Accepted to CVPRW 2025, Workshop on Benchmarking and Expanding AI Multimodal Approaches

LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression

Souvik Kundu, Anahita Bhiwandiwalla, Sungduk Yu, Phillip Howard, Tiep Le, Sharath Nittur Sridhar, David Cobbley, Hao Kang, Vasudev Lal

机构 * Intel Labs（英特尔实验室） ； Georgia Institute of Technology（佐治亚理工学院） 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)；LLaVA(abstract)；visual reasoning(abstract)；分类 cs.CV、cs.AI

Comments This work has been accepted to NAACL 2025

Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learning

Rui Zhao, Qirui Yuan, Jinyu Li, Haofeng Hu, Yun Li, Chengyuan Zheng, Fei Gao

机构 * University of YYY（YYY大学） ； School of ZZZ（ZZZ学院） ； Institute of WWW（WWW研究院） 纠错

专题命中 视觉问答 ：MLLM(title,abstract)；visual question answering(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

PitVQA++: Vector Matrix-Low-Rank Adaptation for Open-Ended Visual Question Answering in Pituitary Surgery

Runlong He, Danyal Z. Khan, Evangelos B. Mazomenos, Hani J. Marcus, Danail Stoyanov, Matthew J. Clarkson, Mobarakol Islam

机构 * University College London（伦敦大学学院） ； UCL Hawkes Institute（伦敦大学学院霍克斯研究所） ； National Hospital for Neurology and Neurosurgery（国立神经学与神经外科医院） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)；vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments 9 pages

Efficient Few-Shot Continual Learning in Vision-Language Models

Aristeidis Panos, Rahaf Aljundi, Daniel Olmeda Reino, Richard E. Turner

机构 * University of Cambridge（剑桥大学） 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)；VLM(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answering

Weixi Weng, Jieming Zhu, Xiaojun Meng, Hao Zhang, Rui Zhang, Chun Yuan

机构 * Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院） ； Huawei Noah's Ark Lab（华为诺亚方舟实验室） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)；multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.LG

DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding

Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, Chong Ruan

机构 * DeepSeek-AI（深度求索科技（DeepSeek）） 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)；visual question answering(abstract)；grounding(abstract)；分类 cs.CV、cs.AI

Improving Medical Diagnostics with Vision-Language Models: Convex Hull-Based Uncertainty Analysis

Ferhat Ozgur Catak, Murat Kuzlu, Taylor Patrick

机构 * University of Stavanger（斯塔万格大学） ； Old Dominion University（奥多明尼昂大学） 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)；VLM(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments 15 pages

LIME: Less Is More for MLLM Evaluation

King Zhu, Qianbo Zang, Shian Jia, Siwei Wu, Feiteng Fang, Yizhi Li, Shawn Gavin, Tuney Zheng, Jiawei Guo, Bo Li, Haoning Wu, Xingwei Qu, Jian Yang, Zachary Liu, Xiang Yue, J. H. Liu, Chenghua Lin, Min Yang, Shiwen Ni, Wenhao Huang, Ge Zhang

专题命中 视觉问答 ：MLLM(title,abstract)；visual question answering(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering

Xinyue Hu, Lin Gu, Qiyuan An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, Ronald M. Summers, Yingying Zhu

专题命中 视觉问答 ：visual question answering(title,abstract)；vision-language model(abstract)；vision language model(abstract)；分类 cs.CV、cs.LG

Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQA

Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chen Kuo, Xinze Guan, Xin Eric Wang

专题命中 视觉问答 ：multimodal large language model(title,abstract)；visual reasoning(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments ACL 2024

Advancing High Resolution Vision-Language Models in Biomedicine

Zekai Chen, Arda Pekis, Kevin Brown

专题命中 视觉问答 ：vision-language model(title,abstract)；LLaVA(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments 15 pages

FreeVA: Offline MLLM as Training-Free Video Assistant

Wenhao Wu

专题命中 视觉问答 ：MLLM(title,abstract)；LLaVA(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments Preprint. Work in progress

Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving

Akshay Gopalkrishnan, Ross Greer, Mohan Trivedi

专题命中 视觉问答 ：vision-language model(title,abstract)；vision language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments 9 pages, 3 figures, Accepted at CVPR 2024 Vision and Language for Autonomous Driving and Robotics Workshop

VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models

Zihao Zhu, Mingda Zhang, Shaokui Wei, Bingzhe Wu, Baoyuan Wu

专题命中 视觉问答 ：multimodal large language model(title,abstract)；visual question answering(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted to ICLR 2024

Physically Grounded Vision-Language Models for Robotic Manipulation

Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, Dorsa Sadigh

专题命中 视觉问答 ：vision-language model(title,abstract)；VLM(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Updated version for ICRA 2024

Uncertainty-Aware Evaluation for Vision-Language Models

Vasily Kostumov, Bulat Nutfullin, Oleg Pilipenko, Eugene Ilyushin

专题命中 视觉问答 ：vision-language model(title,abstract)；LLaVA(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI