MFE-ETP：面向多模态基础模型的具身任务规划综合评估基准
MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation Models on Embodied Task Planning
AI总结：
本研究针对多模态基础模型在具身任务规划领域的评估需求，构建含四项核心能力的评估框架，提出MFE-ETP基准与自动评估平台，测试发现现有先进模型远不及人类水平。
摘要：
近年来，多模态基础模型（Multi-modal Foundation Models, MFMs）与具身人工智能（Embodied Artificial Intelligence, EAI）正以前所未有的速度并行发展，二者的融合受到AI研究界的广泛关注。本研究尝试对多模态基础模型在具身任务规划上的表现开展深入全面的评估，旨在揭示其在该领域的能力与局限。为此，我们首先结合具身任务规划的特性构建了一套系统性评估框架，涵盖多模态基础模型的四项核心能力：物体理解、时空感知、任务理解与具身推理。随后，我们提出了名为MFE-ETP的全新基准，该基准具备复杂多变的任务场景、典型且多样的任务类型、难度各异的任务实例，以及从多类具身问答到具身任务推理的丰富测试用例类型。最后，我们提供了一个简单易用的自动评估平台，可支持多款多模态基础模型在该基准上的自动化测试。借助该基准与评估平台，我们对多款当前最优（state-of-the-art）的多模态基础模型进行了评估，发现其表现远落后于人类水平。MFE-ETP是一个高质量、大规模且具备挑战性、贴合真实世界任务的基准。
英文摘要：
In recent years, Multi-modal Foundation Models (MFMs) and Embodied Artificial Intelligence (EAI) have been advancing side by side at an unprecedented pace. The integration of the two has garnered significant attention from the AI research community. In this work, we attempt to provide an in-depth and comprehensive evaluation of the performance of MFM s on embodied task planning, aiming to shed light on their capabilities and limitations in this domain. To this end, based on the characteristics of embodied task planning, we first develop a systematic evaluation framework, which encapsulates four crucial capabilities of MFMs: object understanding, spatio-temporal perception, task understanding, and embodied reasoning. Following this, we propose a new benchmark, named MFE-ETP, characterized its complex and variable task scenarios, typical yet diverse task types, task instances of varying difficulties, and rich test case types ranging from multiple embodied question answering to embodied task reasoning. Finally, we offer a simple and easy-to-use automatic evaluation platform that enables the automated testing of multiple MFMs on the proposed benchmark. Using the benchmark and evaluation platform, we evaluated several state-of-the-art MFMs and found that they significantly lag behind human-level performance. The MFE-ETP is a high-quality, large-scale, and challenging benchmark relevant to real-world tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
