Understanding the deep semantics of images is essential in the era dominated by social media. However, current research works primarily on the superficial description of images, revealing a notable deficiency in the systematic investigation of the inherent deep semantics. In this work, we introduce DEEPEVAL, a comprehensive benchmark to assess Large Multimodal Models' (LMMs) capacities of visual deep semantics. DEEPEVAL includes human-annotated dataset and three progressive subtasks: fine-grained description selection, in-depth title matching, and deep semantics understanding. Utilizing DEEPEVAL, we evaluate 9 open-source LMMs and GPT-4V(ision). Our evaluation demonstrates a substantial gap between the deep semantic comprehension capabilities of existing LMMs and humans. For example, GPT-4V is 30% behind humans in understanding deep semantics, even though it achieves human-comparable performance in image description. Further analysis reveals that LMM performance on DEEPEVAL varies according to the specific facets of deep semantics explored, indicating the fundamental challenges remaining in developing LMMs.
Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities
Adelaide University（阿德莱德大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）；Amap, Alibaba Group（阿里巴巴集团高德地图）；Beihang University（北京航空航天大学）
Uncovering Entity Identity Confusion in Multimodal Knowledge Editing
New Laboratory of Pattern Recognition (NLPR)（模式识别新实验室）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)（多模态人工智能系统国家重点实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing University of Posts and Telecommunications（北京邮电大学）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）；Huazhong University of Science and Technology（华中科技大学）；Shandong University（山东大学）
MuVaC: A Variational Causal Framework for Multimodal Sarcasm Understanding in Dialogues
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Beihang University（北京航空航天大学）
Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages
Tsinghua University（清华大学）；Zhihu Inc.（知乎公司）；ModelBest Inc.（模贝斯特公司）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Renmin University of China（中国人民大学）
MedQA-MM: Shortcuts Behind Medical Visual Reasoning
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Yale University（耶鲁大学）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；VA Bedford Healthcare System（贝斯以色列女执事医疗中心贝德福德分部）；Qingdao Medical College of Qingdao University（青岛大学青岛医学院）；Yale School of Medicine（耶鲁医学院）；National University Hospital, Singapore（新加坡国立大学医院）；Zhejiang University（浙江大学）；Stanford University（斯坦福大学）
