多模态大语言模型流程图理解能力的首次多维评估
First Multi-Dimensional Evaluation of Flowchart Comprehension for Multimodal Large Language Models
AI总结：
针对多模态大语言模型在流程图任务上缺乏全面评估的问题，提出首个多维评估方法FlowCE，涵盖推理、定位识别、信息提取、逻辑验证和总结能力，实验显示GPT4o得分56.63，开源模型Phi-3-Vision最佳得49.97。
摘要：
随着多模态大语言模型（MLLMs）技术的发展，其通用能力日益强大。为了评估MLLMs的各种能力，涌现出众多评估体系。但目前仍缺乏一种全面的方法来评估MLLMs在与流程图相关的任务中的表现，而流程图在日常工作和生活中非常重要。我们提出了首个全面的方法FlowCE，用于从多个维度评估MLLMs在流程图相关任务中的能力。它涵盖了对MLLMs在流程图上的推理、定位识别、信息提取、逻辑验证和总结能力的评估。然而，我们发现即使是GPT4o模型也仅获得56.63分。在开源模型中，Phi-3-Vision取得了最高分49.97。我们希望FlowCE能够为未来基于流程图的MLLMs任务研究做出贡献。项目链接：https://github.com/360AILABNLP/FlowCE
英文摘要：
With the development of Multimodal Large Language Models (MLLMs) technology, its general capabilities are increasingly powerful. To evaluate the various abilities of MLLMs, numerous evaluation systems have emerged. But now there is still a lack of a comprehensive method to evaluate MLLMs in the tasks related to flowcharts, which are very important in daily life and work. We propose the first comprehensive method, FlowCE, to assess MLLMs across various dimensions for tasks related to flowcharts. It encompasses evaluating MLLMs' abilities in Reasoning, Localization Recognition, Information Extraction, Logical Verification, and Summarization on flowcharts. However, we find that even the GPT4o model achieves only a score of 56.63. Among open-source models, Phi-3-Vision obtained the highest score of 49.97. We hope that FlowCE can contribute to future research on MLLMs for tasks based on flowcharts. \url{https://github.com/360AILABNLP/FlowCE}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
