Generating natural and meaningful responses to communicate with multi-modal human inputs is a fundamental capability of Large Vision-Language Models(LVLMs). While current open-source LVLMs demonstrate promising performance in simplified scenarios such as single-turn single-image input, they fall short in real-world conversation scenarios such as following instructions in a long context history with multi-turn and multi-images. Existing LVLM benchmarks primarily focus on single-choice questions or short-form responses, which do not adequately assess the capabilities of LVLMs in real-world human-AI interaction applications. Therefore, we introduce MMDU, a comprehensive benchmark, and MMDU-45k, a large-scale instruction tuning dataset, designed to evaluate and improve LVLMs' abilities in multi-turn and multi-image conversations. We employ the clustering algorithm to ffnd the relevant images and textual descriptions from the open-source Wikipedia and construct the question-answer pairs by human annotators with the assistance of the GPT-4o model. MMDU has a maximum of 18k image+text tokens, 20 images, and 27 turns, which is at least 5x longer than previous benchmarks and poses challenges to current LVLMs. Our in-depth analysis of 15 representative LVLMs using MMDU reveals that open-source LVLMs lag behind closed-source counterparts due to limited conversational instruction tuning data. We demonstrate that ffne-tuning open-source LVLMs on MMDU-45k signiffcantly address this gap, generating longer and more accurate conversations, and improving scores on MMDU and existing benchmarks (MMStar: +1.1%, MathVista: +1.5%, ChartQA:+1.2%). Our contributions pave the way for bridging the gap between current LVLM models and real-world application demands. This project is available at https://github.com/Liuziyu77/MMDU.
SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Kuaishou Technology(快手科技);Zhengzhou University(郑州大学);ShanghaiTech University(上海科技大学);Peng Cheng Laboratory(鹏城实验室)
Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
School of Computer Science and Engineering, Southeast University, China(东南大学计算机科学与工程学院,中国);Alibaba Group(阿里巴巴集团);School of Computer Science and Engineering, School of Intelligence Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China(东南大学计算机科学与工程学院、智能科学与工程学院以及新一代人工智能技术及其交叉应用关键实验室,中国);Wangxuan Institute of Computer Technology, National Key Laboratory for Multimedia Information Processing, Peking University, China(北京大学王轩计算机技术研究所、多媒体信息处理国家重点实验室,中国);University of Copenhagen, Denmark(丹麦哥本哈根大学)
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The University of Hong Kong(香港大学);The University of Adelaide(阿德莱德大学);Zhejiang University(浙江大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions
The University of Hong Kong(香港大学);Carnegie Mellon University(卡内基梅隆大学);Johns Hopkins University(约翰霍普金斯大学);University of California Irvine(加州大学尔湾分校);Stanford University(斯坦福大学)
Rethinking Token Reduction for Large Vision-Language Models
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院);School of Software Technology, Zhejiang University(浙江大学软件学院);National University of Singapore(新加坡国立大学);Alibaba Group(阿里巴巴集团)
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学);Nanjing University(南京大学);Fudan University(复旦大学);Nanyang Technological University(南洋理工大学)
Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis
Faculty of Dentistry, The University of Hong Kong(香港大学牙科学院);The Hong Kong University of Science and Technology (GZ)(香港科学与技术大学);National University of Singapore(新加坡国立大学);CVTE;Sun Yat-sen University(孙中山大学);Department of Diagnostic Radiology, The University of Hong Kong(香港大学放射科);Imaging and Interventional Radiology, Faculty of Medicine, The Chinese University of Hong Kong(香港中文大学医学院影像与介入放射科);School of Computer Science, Peking University(北京大学计算机科学系)