arXiv 2402.155272024-02-21cs.CLcs.AIcs.CV

PCA-Bench：评估多模态大语言模型在感知-认知-行动链中的表现

PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain

Peking University（北京大学）；Alibaba Group（阿里巴巴集团）

Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, Baobao Chang

备注：: Code and Data released at https://github.com/pkunlp-icler/PCA-EVAL. Leaderboard at: https://docs.qq.com/sheet/DVUd4WUpGRHRqUnNV. This article supersedes its workshop version arxiv: 2310.02071. arXiv admin note: text overlap with arXiv:2310.02071

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出PCA-Bench多模态决策基准及PCA-Eval评估协议，揭示开源与专有MLLMs的性能差距，并通过EIE框架自动合成指令微调数据，有效提升开源模型决策能力。

摘要：

我们提出PCA-Bench，一个用于评估多模态大语言模型（MLLMs）综合能力的多模态决策基准。与以往聚焦简单任务和单一模型能力的基准不同，PCA-Bench引入了三个复杂场景：自动驾驶、家用机器人和开放世界游戏。在给定任务指令和多样化情境下，模型需要在推理链中无缝整合感知、认知和行动等多种能力，以做出准确决策。此外，PCA-Bench具备错误定位能力，能够细致检查模型在感知、知识或推理等方面的不准确性，从而提升部署MLLMs的可靠性。为了在评估中平衡准确性与效率，我们提出了自动评估协议PCA-Eval，并评估了10个主流MLLMs。结果揭示了开源模型与GPT-4 Vision等强大专有模型之间存在显著的性能差距。为解决这一问题，我们引入了Embodied-Instruction-Evolution（EIE），一个用于在多模态具身环境中自动合成指令微调示例的框架。EIE在PCA-Bench中生成了7,510个训练示例，并提升了开源MLLMs的性能，偶尔超过GPT-4 Vision（决策准确率提升3%），从而验证了EIE的有效性。我们的研究结果表明，像GPT4-Vision这样强大的MLLMs在具身智能体决策方面展现出潜力，为MLLM研究开辟了新途径。

英文摘要：

We present PCA-Bench, a multimodal decision-making benchmark for evaluating the integrated capabilities of Multimodal Large Language Models (MLLMs). Departing from previous benchmarks focusing on simplistic tasks and individual model capability, PCA-Bench introduces three complex scenarios: autonomous driving, domestic robotics, and open-world games. Given task instructions and diverse contexts, the model is required to seamlessly integrate multiple capabilities of Perception, Cognition, and Action in a reasoning chain to make accurate decisions. Moreover, PCA-Bench features error localization capabilities, scrutinizing model inaccuracies in areas such as perception, knowledge, or reasoning. This enhances the reliability of deploying MLLMs. To balance accuracy and efficiency in evaluation, we propose PCA-Eval, an automatic evaluation protocol, and assess 10 prevalent MLLMs. The results reveal significant performance disparities between open-source models and powerful proprietary models like GPT-4 Vision. To address this, we introduce Embodied-Instruction-Evolution (EIE), an automatic framework for synthesizing instruction tuning examples in multimodal embodied environments. EIE generates 7,510 training examples in PCA-Bench and enhances the performance of open-source MLLMs, occasionally surpassing GPT-4 Vision (+3\% in decision accuracy), thereby validating the effectiveness of EIE. Our findings suggest that robust MLLMs like GPT4-Vision show promise for decision-making in embodied agents, opening new avenues for MLLM research.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。