ChEF：用于多模态大语言模型标准化评估的综合评估框架
ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
AI总结：
针对多模态大语言模型缺乏标准化整体评估框架的问题，提出含四个模块化组件的ChEF评估框架，设计6种能力评估方案，完成9个主流模型的大规模评估并公开发布工具包。
摘要：
多模态大语言模型（MLLMs）在视觉内容交互方面展现出令人瞩目的能力，拥有众多潜在的下游任务。然而，尽管已有一系列基准被提出，由于缺乏标准化的整体性评估框架，MLLMs的能力与局限性仍未得到全面理解。为此，我们提出了首个综合评估框架（Comprehensive Evaluation Framework，简称ChEF），该框架能够对每个MLLM进行整体性剖析，并对不同MLLM开展公平比较。首先，我们将ChEF构建为四个模块化组件，分别是作为可扩展多模态数据集的Scenario（场景）、作为灵活指令检索公式的Instruction（指令）、作为可靠问答策略的Inferencer（推理器），以及作为具有指示性的特定任务评分函数的Metric（指标）。基于这些组件，ChEF可在标准化框架内支持多样化评估，且可通过设计新的Recipe（方案，即对这四个组件的系统性选择）构建新的评估。值得注意的是，当前的MLLM基准均可轻松归纳为ChEF的方案。其次，我们引入了6种新方案，用以量化胜任的MLLM作为可执行真实世界多模态交互的可靠智能体所需具备的理想能力（即期望特性，包括校准、上下文学习、指令遵循、语言性能、幻觉和鲁棒性）。第三，我们在9个场景和6项期望特性上对9个主流MLLM开展了大规模评估。我们的评估总结出20余项有价值的发现，涉及MLLM在不同场景间的泛化能力，以及多模态交互所需的MLLM综合能力。我们将公开发布所有详细实现以供进一步分析，同时提供一个易用的模块化工具包，用于集成新的方案和模型，使ChEF能够成为服务于MLLM社区的可持续发展的评估框架。
英文摘要：
Multimodal Large Language Models (MLLMs) have shown impressive abilities in interacting with visual content with myriad potential downstream tasks. However, even though a list of benchmarks has been proposed, the capabilities and limitations of MLLMs are still not comprehensively understood, due to a lack of a standardized and holistic evaluation framework. To this end, we present the first Comprehensive Evaluation Framework (ChEF) that can holistically profile each MLLM and fairly compare different MLLMs. First, we structure ChEF as four modular components, i.e., Scenario as scalable multimodal datasets, Instruction as flexible instruction retrieving formulae, Inferencer as reliable question answering strategies, and Metric as indicative task-specific score functions. Based on them, ChEF facilitates versatile evaluations in a standardized framework, and new evaluations can be built by designing new Recipes (systematic selection of these four components). Notably, current MLLM benchmarks can be readily summarized as recipes of ChEF. Second, we introduce 6 new recipes to quantify competent MLLMs' desired capabilities (or called desiderata, i.e., calibration, in-context learning, instruction following, language performance, hallucination, and robustness) as reliable agents that can perform real-world multimodal interactions. Third, we conduct a large-scale evaluation of 9 prominent MLLMs on 9 scenarios and 6 desiderata. Our evaluation summarized over 20 valuable observations concerning the generalizability of MLLMs across various scenarios and the composite capability of MLLMs required for multimodal interactions. We will publicly release all the detailed implementations for further analysis, as well as an easy-to-use modular toolkit for the integration of new recipes and models, so that ChEF can be a growing evaluation framework for the MLLM community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
