arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.11833cs.CVcs.AIcs.LG
收藏

MMDU:面向大视觉语言模型的多轮多图像对话理解基准与指令微调数据集

MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs

SJTU(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);MThreads, Inc.(MThreads公司);CUHK(香港中文大学);CPII under InnoHK(InnoHK下属CPII机构)

Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, Jiaqi Wang

AI总结:

针对现有LVLM在多轮多图像真实对话场景表现不足、基准评估不充分的问题,提出MMDU基准与MMDU-45k指令微调数据集,验证了用其微调可提升开源LVLM的对话能力与多基准得分。

摘要:

针对多模态人类输入生成自然且有意义的响应以开展交互,是大视觉语言模型(Large Vision-Language Models, LVLMs)的一项基础能力。当前开源LVLM在单轮单图像输入等简化场景中表现出可观性能,但在长上下文历史中遵循指令、处理多轮多图像的真实对话场景中仍存在不足。现有LVLM基准主要聚焦选择题或短回答,无法充分评估LVLM在真实人机交互应用中的能力。\n因此,我们提出了综合性基准MMDU,以及大规模指令微调数据集MMDU-45k,旨在评估并提升LVLM的多轮多图像对话能力。我们采用聚类算法从开源Wikipedia中筛选相关图像与文本描述,并在GPT-4o模型辅助下由人工标注者构建问答对。MMDU的图像+文本token数最高达18k,最多包含20张图像、27轮对话,长度至少是此前基准的5倍,对当前LVLM构成了挑战。\n我们使用MMDU对15个代表性LVLM开展深入分析,结果显示,由于对话指令微调数据有限,开源LVLM的表现落后于闭源同类模型。我们证明,在MMDU-45k上微调开源LVLM可显著缩小这一差距,能生成更长、更准确的对话内容,同时提升了在MMDU及现有基准上的得分(MMStar:+1.1%,MathVista:+1.5%,ChartQA:+1.2%)。我们的研究为弥合当前LVLM模型与真实应用需求之间的差距奠定了基础。该项目可在https://github.com/Liuziyu77/MMDU 获取。

英文摘要:

Generating natural and meaningful responses to communicate with multi-modal human inputs is a fundamental capability of Large Vision-Language Models(LVLMs). While current open-source LVLMs demonstrate promising performance in simplified scenarios such as single-turn single-image input, they fall short in real-world conversation scenarios such as following instructions in a long context history with multi-turn and multi-images. Existing LVLM benchmarks primarily focus on single-choice questions or short-form responses, which do not adequately assess the capabilities of LVLMs in real-world human-AI interaction applications. Therefore, we introduce MMDU, a comprehensive benchmark, and MMDU-45k, a large-scale instruction tuning dataset, designed to evaluate and improve LVLMs' abilities in multi-turn and multi-image conversations. We employ the clustering algorithm to ffnd the relevant images and textual descriptions from the open-source Wikipedia and construct the question-answer pairs by human annotators with the assistance of the GPT-4o model. MMDU has a maximum of 18k image+text tokens, 20 images, and 27 turns, which is at least 5x longer than previous benchmarks and poses challenges to current LVLMs. Our in-depth analysis of 15 representative LVLMs using MMDU reveals that open-source LVLMs lag behind closed-source counterparts due to limited conversational instruction tuning data. We demonstrate that ffne-tuning open-source LVLMs on MMDU-45k signiffcantly address this gap, generating longer and more accurate conversations, and improving scores on MMDU and existing benchmarks (MMStar: +1.1%, MathVista: +1.5%, ChartQA:+1.2%). Our contributions pave the way for bridging the gap between current LVLM models and real-world application demands. This project is available at https://github.com/Liuziyu77/MMDU.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑