Large vision-language models (LVLMs) have recently achieved rapid progress, sparking numerous studies to evaluate their multi-modal capabilities. However, we dig into current evaluation works and identify two primary issues: 1) Visual content is unnecessary for many samples. The answers can be directly inferred from the questions and options, or the world knowledge embedded in LLMs. This phenomenon is prevalent across current benchmarks. For instance, GeminiPro achieves 42.9% on the MMMU benchmark without any visual input, and outperforms the random choice baseline across six benchmarks over 24% on average. 2) Unintentional data leakage exists in LLM and LVLM training. LLM and LVLM could still answer some visual-necessary questions without visual content, indicating the memorizing of these samples within large-scale training data. For example, Sphinx-X-MoE gets 43.6% on MMMU without accessing images, surpassing its LLM backbone with 17.9%. Both problems lead to misjudgments of actual multi-modal gains and potentially misguide the study of LVLM. To this end, we present MMStar, an elite vision-indispensable multi-modal benchmark comprising 1,500 samples meticulously selected by humans. MMStar benchmarks 6 core capabilities and 18 detailed axes, aiming to evaluate LVLMs' multi-modal capacities with carefully balanced and purified samples. These samples are first roughly selected from current benchmarks with an automated pipeline, human review is then involved to ensure each curated sample exhibits visual dependency, minimal data leakage, and requires advanced multi-modal capabilities. Moreover, two metrics are developed to measure data leakage and actual performance gain in multi-modal training. We evaluate 16 leading LVLMs on MMStar to assess their multi-modal capabilities, and on 7 benchmarks with the proposed metrics to investigate their data leakage and actual multi-modal gain.
Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University of Oxford(牛津大学);Xi’an Jiaotong University(西安交通大学);The Hong Kong University of Science and Technology(香港科学与技术大学);City University of Hong Kong(香港城市大学);Beijing University of Technology(北京理工大学);Duke University(杜克大学);X-Humanoid Project(X-Humanoid 项目)
WMVLM: Evaluating Diffusion Model Image Watermarking via Vision-Language Models
University of Science and Technology of China(中国科学技术大学);Anhui Province Key Laboratory of Digital Security(安徽省数字安全重点实验室);National University of Singapore(新加坡国立大学)
GeoArena: Evaluating Open-World Geographic Reasoning in Large Vision-Language Models
Department of Data Science, City University of Hong Kong(香港城市大学数据科学系);Department of Computer Sciences, University of Wisconsin-Madison(威斯康星大学麦迪逊分校计算机科学系)
Evaluating Vision-Language Models as a Zero-Shot Learning Alternative to You Only Look Once and Optical Character Recognition for Nigerian License Plate Recognition
Ahmadu Bello University(艾哈迈杜·贝罗大学);Bayero University Kano(卡诺贝罗大学)
Automated Report-Derived Oncology VQA Benchmark for Evaluating Vision-Language Models on 3D Medical Imaging
UCSF–UC Berkeley Joint Graduate Program in Bioengineering(UCSF-伯克利生物工程联合研究生项目);Department of Radiology, UCSF(UCSF放射科);Department of Radiation Oncology, UCSF(UCSF放射肿瘤科)
Visual Evidence Under Cross-Examination: Evaluating and Controlling Decision-Level Evidence Use in Vision-Language Models
Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences(中国科学院空间应用工程与技术中心);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学)
MVTamperBench: Evaluating Robustness of Vision-Language Models
Liverpool John Moores University(利物浦约翰·穆里斯大学);Birla Institute of Technology(比拉理工学院);Christ University(基督大学);Columbia University(哥伦比亚大学);New York University(纽约大学);University of Washington(华盛顿大学);Hanyang University(翰阳大学)
What's left can't be right -- The remaining positional incompetence of contrastive vision-language models
SFI Centre for Research Training in Machine Learning at University College Dublin(都柏林大学学院SFI机器学习研究培训中心);School of Computer Science, University College Dublin(都柏林大学学院计算机科学学院);School of Computer Science and Statistics, Trinity College Dublin(都柏林圣三一学院计算机科学与统计学院)
VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service
The Hong Kong University of Science and Technology(香港科学与技术大学);Tongji University(同济大学);Beijing Jiaotong University(北京交通大学);Huawei(华为);Tsinghua University(清华大学);University of Chinese Academy of Sciences(中国科学院大学)
Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanese
Turing Inc.(图灵公司);Keio Research Institute at SFC(庆应义塾大学SFC研究所);University of Tsukuba(筑波大学);National Institute of Technology, Kisarazu College(木更津工业高等专门学校);Tokyo Institute of Technology(东京工业大学)