MM-Vet, with open-ended vision-language questions targeting at evaluating integrated capabilities, has become one of the most popular benchmarks for large multimodal model evaluation. MM-Vet assesses six core vision-language (VL) capabilities: recognition, knowledge, spatial awareness, language generation, OCR, and math. However, its question format is restricted to single image-text pairs, lacking the interleaved image and text sequences prevalent in real-world scenarios. To address this limitation, we introduce MM-Vet v2, which includes a new VL capability called "image-text sequence understanding", evaluating models' ability to process VL sequences. Furthermore, we maintain the high quality of evaluation samples while further expanding the evaluation set size. Using MM-Vet v2 to benchmark large multimodal models, we found that Claude 3.5 Sonnet is the best model with a score of 71.8, slightly outperforming GPT-4o which scored 71.0. Among open-weight models, InternVL2-Llama3-76B leads with a score of 68.4. The code, data, and leaderboard are accessible at https://github.com/yuweihao/MM-Vet.
MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
Hong Kong Baptist University(香港 Baptist 大学);Beijing Normal-Hong Kong Baptist University(北京师范大学-香港 Baptist 大学);National University of Singapore(新加坡国立大学);Beijing Normal University(北京师范大学);Shanghai Jiao Tong University(上海交通大学)
Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);iFLYTEK AI Research(iFLYTEK人工智能研究院);Hong Kong University of Science and Technology(香港科学与技术大学)
ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
Nankai University(南开大学);Shanghai Innovation Institute(上海创新研究院);University of Science and Technology of China(中国科学技术大学);Wuhan University(武汉大学);Shanghai AI Laboratory(上海人工智能实验室)
Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
Hacettepe University(哈杰泰佩大学);Aselsan Research(阿塞尔桑研究院);Koç University(科克大学);Amsterdam UMC, University of Amsterdam(阿姆斯特丹大学医学中心,阿姆斯特丹大学);Amsterdam Public Health(阿姆斯特丹公共卫生研究院)
TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing Prompts
OpenGV Lab(OpenGV实验室);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院)
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
ZIP Lab, Zhejiang University(浙江大学 ZIP 实验室);South China University of Technology(华南理工大学);Pazhou Lab(琶洲实验室);Key Laboratory of Big Data and Intelligent Robot, Ministry of Education(大数据与智能机器人教育部重点实验室)
Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室);Wuhan AI Research, Wuhan, China(武汉人工智能研究所)
Leveraging ChatGPT's Multimodal Vision Capabilities to Rank Satellite Images by Poverty Level: Advancing Tools for Social Science Research
Center for Applied Intelligent Systems Research (CAISR), Halmstad University, Sweden(应用智能系统研究中心(CAISR),哈马斯特德大学,瑞典);Department of Human Geography, Lund University, Sweden(人类地理系,吕勒奥大学,瑞典);Department of Earth and Environmental Sciences, Lund University, Sweden(地球与环境科学系,吕勒奥大学,瑞典)
Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning
Zhejiang University(浙江大学);Fudan University(复旦大学);Soochow University(苏州大学);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学)
Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models
Fudan University(复旦大学);Meituan(美团);Shanghai Key Lab of Intell. Info. Processing(上海智能信息处理重点实验室);Shanghai Collaborative Innovation Center on Intelligent Visual Computing(上海智能视觉计算协同创新中心)
X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Future Technology, University of Chinese Academy of Sciences(中国科学院大学未来技术学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Maya: An Instruction Finetuned Multilingual Multimodal Model
Cohere For AI Community(Cohere 人工智能社区);Indiana University Bloomington(印第安纳大学伯明顿分校);Imperial College London(伦敦帝国学院);Georgia Institute of Technology(佐治亚理工学院);The Alan Turing Institute(艾伦·图灵研究所);Bangladesh University of Engineering and Technology(孟加拉工程技术大学);University of Pennsylvania(宾夕法尼亚大学);IIT Bombay(印度理工学院孟买分校);TU Darmstadt(达姆施塔特工业大学);IIT Dhanbad(印度理工学院丹巴德分校);MBZUAI(穆罕默德·本·扎耶德人工智能大学);Cisco Meraki(思科 Meraki);Articul8 AI(Articul8 人工智能公司);Capital One(第一资本金融公司)