Large Vision-Language Models (LVLMs) have demonstrated outstanding performance across various multimodal tasks. However, they suffer from a problem known as language prior, where responses are generated based solely on textual patterns while disregarding image information. Addressing the issue of language prior is crucial, as it can lead to undesirable biases or hallucinations when dealing with images that are out of training distribution. Despite its importance, current methods for accurately measuring language priors in LVLMs are poorly studied. Although existing benchmarks based on counterfactual or out-of-distribution images can partially be used to measure language priors, they fail to disentangle language priors from other confounding factors. To this end, we propose a new benchmark called VLind-Bench, which is the first benchmark specifically designed to measure the language priors, or blindness, of LVLMs. It not only includes tests on counterfactual images to assess language priors but also involves a series of tests to evaluate more basic capabilities such as commonsense knowledge, visual perception, and commonsense biases. For each instance in our benchmark, we ensure that all these basic tests are passed before evaluating the language priors, thereby minimizing the influence of other factors on the assessment. The evaluation and analysis of recent LVLMs in our benchmark reveal that almost all models exhibit a significant reliance on language priors, presenting a strong challenge in the field.
Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）
Measuring the Measurers: Quality Evaluation of Hallucination Benchmarks for Large Vision-Language Models
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, and also with University of Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院，以及中国科学院大学）
Measuring what Matters: Construct Validity in Large Language Model Benchmarks
University of Oxford（牛津大学）；EPFL（苏黎世联邦理工学院）；Weizenbaum Institute Berlin（柏林Weizenbaum研究所）；Technical University Munich（慕尼黑技术大学）；Centre for Digital Governance, Hertie School（赫尔姆霍兹学院数字治理中心）；Stanford University（斯坦福大学）；UK AI Security Institute（英国人工智能安全研究所）；SomosNLP；Universdad Politécnica de Madrid（马德里理工大学）；Yale University（耶鲁大学）；Allen Institute for AI（人工智能研究所）；University of Washington（华盛顿大学）；Meedan；UC Berkeley（加州大学伯克利分校）
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Gray Swan AI（格雷天鹅人工智能公司）；UK AI Security Institute（英国人工智能安全研究所）；EPFL（洛桑联邦理工学院）；Center for AI Safety（人工智能安全中心）；Carnegie Mellon University（卡内基梅隆大学）；University of Oxford（牛津大学）
ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
Artificial Intelligence and Intelligent Information Systems, University of Trier（人工智能与智能信息系统，特里尔大学）；German Research Center for Artificial Intelligence (DFKI), Trier（德国人工智能研究中心（DFKI），特里尔）
Measuring the Unspoken: A Disentanglement Model and Benchmark for Psychological Analysis in the Wild
College of Computer Science, National University of Defense Technology（计算机科学学院，国防科技大学）；Shien-Ming Wu School of Intelligent Engineering, South China University of Technology（智能工程学院，华南理工大学）
MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors
Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；ETH AI Center（ETH人工智能中心）；Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany（达姆施塔特技术大学计算机科学系）；Professorship for Learning Sciences and Higher Education, ETH Zurich（苏黎世联邦理工学院学习科学与高等教育教授职位）
