arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.16641cs.CV
收藏

MultiVerse:一个用于评估大型视觉语言模型的多轮对话基准

MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models

KAIST(韩国科学技术院);WHU(华中科技大学);NAVER;CMU(卡内基梅隆大学)

Young-Jun Lee, Byung-Kwan Lee, Jianshu Zhang, Yechan Hwang, Byungsoo Ko, Han-Gyu Kim, Dongyu Yao, Xuankun Rong, Eojin Joo, Seung-Ho Han, Bowon Ko, Ho-Jin Choi

AI总结:

本文提出 MultiVerse 多轮对话基准,基于12个VLM基准构建647段对话,并用 GPT-4o 从37个方面自动评估18个模型,揭示现有模型在复杂多轮交互中的不足。

摘要:

视觉语言模型(VLM)在单轮基准测试中展现出令人瞩目的能力,但现实世界应用通常需要更复杂的多轮对话。现有多轮数据集(例如 MMDU、ConvBench)只能部分涵盖用户所遇对话场景的广度与深度。在本工作中,我们提出 MultiVerse,这是一个新颖的多轮对话基准,包含 647 段对话,每段平均四轮,来源于 12 个流行的 VLM 评估基准,具有多样性。MultiVerse 包含 484 个任务和 484 个交互目标,覆盖从事实知识、感知到数学和编程等高级推理任务的广泛主题。为促进稳健评估,我们提出一种基于检查表的评估方法,利用 GPT-4o 作为自动评估器,从 37 个关键方面衡量性能,包括感知准确性、语言清晰性和事实正确性。我们在 MultiVerse 上评估了 18 个 VLM,结果表明,即使是最强的模型(例如 GPT-4o)在复杂多轮对话中也仅达到 50% 的成功率,突显了该数据集的挑战性。值得注意的是,我们发现提供完整对话上下文会显著提升较小或较弱模型的性能,强调了上下文内学习的重要性。我们相信 MultiVerse 是评估 VLM 多轮交互能力的一幅全景图。

英文摘要:

Vision-and-Language Models (VLMs) have shown impressive capabilities on single-turn benchmarks, yet real-world applications often demand more intricate multi-turn dialogues. Existing multi-turn datasets (e.g, MMDU, ConvBench) only partially capture the breadth and depth of conversational scenarios encountered by users. In this work, we introduce MultiVerse, a novel multi-turn conversation benchmark featuring 647 dialogues - each averaging four turns - derived from a diverse set of 12 popular VLM evaluation benchmarks. With 484 tasks and 484 interaction goals, MultiVerse covers a wide range of topics, from factual knowledge and perception to advanced reasoning tasks such as mathematics and coding. To facilitate robust assessment, we propose a checklist-based evaluation method that leverages GPT-4o as the automated evaluator, measuring performance across 37 key aspects, including perceptual accuracy, linguistic clarity, and factual correctness. We evaluate 18 VLMs on MultiVerse, revealing that even the strongest models (e.g., GPT-4o) achieve only a 50% success rate in complex multi-turn conversations, highlighting the dataset's challenging nature. Notably, we find that providing full dialogue context significantly enhances performance for smaller or weaker models, emphasizing the importance of in-context learning. We believe MultiVerse is a landscape of evaluating multi-turn interaction abilities for VLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑