arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09295cs.CLcs.CV
收藏

AlignMMBench:评估大型视觉语言模型中的中文多模态对齐能力

AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models

Tsinghua University(清华大学);Zhipu AI(智谱AI);Peking University(北京大学)

Yuhang Wu, Wenmeng Yu, Yean Cheng, Yan Wang, Xiaohan Zhang, Jiazheng Xu, Ming Ding, Yuxiao Dong

AI总结:

本文提出首个面向中文视觉语境的多模态对齐基准AlignMMBench,包含13项任务和4978个问答对,并开发了超越GPT-4的评估器CritiqueVLM及对齐分数指标,系统评估了代表性VLM的对齐能力。

摘要:

评估大型视觉语言模型(VLMs)的对齐能力对于确定其作为有用助手的有效性至关重要。然而,现有基准主要关注使用非语言方法(如是非题和多项选择题)的基础能力。在本文中,我们通过引入AlignMMBench来弥补这一空白,该基准提供了对对齐能力更细致的评估,并且是首个专门为中文视觉语境设计的基准。该基准从真实世界场景和互联网来源中精心构建,涵盖三个类别下的十三项具体任务,并包括单轮和多轮对话场景。结合提示重写策略,AlignMMBench包含1,054张图像和4,978个问答对。为了促进评估流程,我们开发了CritiqueVLM,这是一个经过规则校准的评估器,其评估能力超过了GPT-4。此外,我们测量了“对齐分数”,这是一种定量指标,旨在评估模型在不同提示下的鲁棒性和稳定性。最后,我们评估了代表性VLM在AlignMMBench上的性能,为不同VLM架构的能力和局限性提供了见解。评估代码和数据可在https://github.com/THUDM/AlignMMBench获取。

英文摘要:

Evaluating the alignment capabilities of large Vision-Language Models (VLMs) is essential for determining their effectiveness as helpful assistants. However, existing benchmarks primarily focus on basic abilities using nonverbal methods, such as yes-no and multiple-choice questions. In this paper, we address this gap by introducing AlignMMBench, which provides more nuanced evaluations of alignment capabilities and is the first benchmark specifically designed for Chinese visual contexts. This benchmark is meticulously curated from real-world scenarios and internet sources, encompassing thirteen specific tasks across three categories, and includes both single-turn and multi-turn dialogue scenarios. Incorporating a prompt rewrite strategy, AlignMMBench encompasses 1,054 images and 4,978 question-answer pairs. To facilitate the evaluation pipeline, we develop CritiqueVLM, a rule-calibrated evaluator that exceeds GPT-4's evaluation ability. Additionally, we measure the "alignment score", a quantitative metric designed to assess the robustness and stability of models across diverse prompts. Finally, we evaluate the performance of representative VLMs on AlignMMBench, offering insights into the capabilities and limitations of different VLM architectures. The evaluation code and data are available at https://github.com/THUDM/AlignMMBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑