DeepTumorVQA: A Hierarchical 3D CT Benchmark for Stage-Wise Evaluation of Medical VLMs and Tool-Augmented Agents
Johns Hopkins University(约翰霍普金斯大学);University of Bologna(博洛尼亚大学);Istanbul Medipol University(伊斯坦布尔梅迪波尔大学);Center for Biomolecular Nanotechnologies, Istituto Italiano di Tecnologia(生物分子纳米技术中心,意大利技术研究院);The First Affiliated Hospital, Sun Yat-Sen University(中山大学第一附属医院);Tongji University(同济大学)
Yixiong Chen, Wenjie Xiao, Pedro R. A. S. Bassi, Boyan Wang, Liang He, Xinze Zhou, Sezgin Er, Ibrahim Ethem Hamamci, Zongwei Zhou, Alan Yuille
Medical vision-language models (VLMs) and AI agents have made significant progress in learning to analyze and reason about clinical images. However, existing medical visual question answering (VQA) benchmarks collapse model capabilities into a single accuracy score, obscuring where and why models fail. We propose DeepTumorVQA, a hierarchical benchmark that follows the multi-stage evidence chain in tumor diagnosis and decomposes 3D CT reasoning into four stages: recognition, measurement, visual reasoning, and medical reasoning. Higher-level questions remain independently scorable, while their ground-truth evidence chains are defined over lower-level primitives. The benchmark contains 476K questions across 42 clinical subtypes on 9,262 3D CT volumes. In addition to a direct reasoning mode for VLMs, DeepTumorVQA provides tool-interaction environments for agent evaluation, where a model can call external tools, including segmentation models, measurement programs, and medical knowledge modules, before answering the question. Evaluating over 30 model configurations, we find that reliable quantitative measurement is the primary bottleneck, making later-stage visual and medical reasoning harder for VLMs, while tool augmentation substantially mitigates this issue. When tools are available, leveraging medical knowledge and tools to reason about medical images becomes a new challenge. We further show that ground-truth step-by-step tool-use traces from DeepTumorVQA can supervise agents and reduce tool-use and reasoning failures. This stage-wise progression from recognition to measurement to visual and medical reasoning provides a concrete roadmap for future medical VLM and AI agent studies. All data and code are released at https://github.com/Schuture/DeepTumorVQA.
Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering
Johns Hopkins University(约翰霍普金斯大学);University of Bologna(博洛尼亚大学);Istanbul Medipol University(伊斯坦布尔梅迪波尔大学);Center for Biomolecular Nanotechnologies, Istituto Italiano di Tecnologia(意大利理工学院生物分子纳米技术中心)
Early Exit and Multi Stage Knowledge Distillation in VLMs for Video Summarization
Department of Computer Science and Engineering, IIT Bombay(印度理工学院班加罗尔计算机科学与工程系);Center of Machine Intelligence and Data Science (C-MInDS), IIT Bombay(印度理工学院班加罗尔人工智能与数据科学中心)
DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Models
Tuebingen AI Center University of Tuebingen(图宾根人工智能中心图宾根大学);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);MIT-IBM Watson AI Lab(麻省理工-IBM沃森人工智能实验室);IBM Research(IBM研究院)
How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study
School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences)(教育部计算功率网络与信息安全部重点实验室,山东计算机科学中心(济南国家超级计算中心),齐鲁工业大学(山东省科学院))
NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models
The University of Manchester(曼彻斯特大学);University of Michigan(密歇根大学);Sheffield Hallam University(谢菲尔德哈勒姆大学);Tufts University(塔夫茨大学);ENSTA, Institut Polytechnique de Paris(巴黎综合理工学院国立高等先进技术学校)
UkrMeme: Multimodal Explainable Persuasion Detection in Ukrainian Memes
Ukrainian Catholic University(乌克兰天主教大学);Johannes Gutenberg University Mainz(美因茨约翰内斯·古滕贝格大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Centre for European Research in Trusted AI (CERTAIN)(欧洲可信人工智能研究中心)
BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models
POSTECH(浦项科技大学);Grad. School of AI, POSTECH(浦项科技大学人工智能学院);Institute for Convergence Research and Education in Advanced Technology, Yonsei University(延世大学尖端技术融合研究教育学院)
Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming
Computer Aided Medical Procedures (CAMP)(计算机辅助医疗程序);TU Munich, Germany(慕尼黑工业大学,德国);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Munich, Germany(慕尼黑,德国);Zhongshan Hospital, Fudan University, China(复旦大学中山医院);The University of Hong Kong, Hongkong, China(香港大学,香港,中国)
Med-StepBench: A Hierarchical Reasoning Framework for Evaluating Hallucinations in Medical Vision-Language Models
AI4LIFE, Hanoi University of Science and Technology, Vietnam(AI4LIFE,越南科学与技术大学);SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, France(SAMOVAR,法国电信南巴黎学院,巴黎理工学院);Military Central Hospital, Vietnam(越南108军中心医院)
Optimizing Vision-Language Consistency via Cross-Layer Regional Attention Alignment
School of Medicine, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)医学院);Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Wave and Machine Intelligence Department, Technology Innovation Institute(技术创新研究院波浪与机器智能部门);University of the Chinese Academy of Sciences(中国科学院大学);McGill University(麦吉尔大学)
ClimateVID -- Social Media Videos Analysis and Challenges Involved
University of Mannheim(曼海姆大学);Technical University of Munich(慕尼黑技术大学);Max-Planck-Institute for Informatics(马克斯·普朗克信息研究所);Saarland Informatics Campus(萨尔兰州信息学校区)
Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs
The Hong Kong Polytechnic University(香港理工大学);Eastern Institute of Technology, Ningbo(宁波东方理工大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Shanghai Jiao Tong University(上海交通大学)