ActPlan-1K: 基准测试视觉语言模型在家庭活动中的流程规划能力
ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities
AI总结：
本研究提出多模态规划基准ActPlan-1K，从多模态和反事实维度评估VLM的家庭活动流程规划能力，发现当前VLM表现仍有不足，并提供自动评估指标助力后续研究。
摘要：
大语言模型（LLMs）凭借强大的推理能力，已被用于处理具身AI任务中的文本任务描述并完成流程规划。然而，关于视觉语言模型（VLMs）在多模态任务输入下的表现研究仍然不足，评估模型对替代任务场景推理能力的反事实规划也未得到充分探索。为了从多模态和反事实两个维度评估规划能力，我们提出了ActPlan-1K。ActPlan-1K是基于ChatGPT和家庭活动模拟器iGibson2构建的多模态规划基准，包含153项活动和1187个实例。每个描述一项活动的实例都配有自然语言任务描述和多张来自模拟器的环境图像，每个实例的黄金规划是针对所提供场景中物体的动作序列。我们在典型VLM上评估了规划的正确性和常识符合度，结果表明当前VLM在为常规活动和反事实活动生成人类水平的流程规划方面仍存在困难。我们还通过微调BLEURT模型提供了自动评估指标，以推动基于该基准的未来研究。
英文摘要：
Large language models~(LLMs) have been adopted to process textual task description and accomplish procedural planning in embodied AI tasks because of their powerful reasoning ability. However, there is still lack of study on how vision language models~(VLMs) behave when multi-modal task inputs are considered. Counterfactual planning that evaluates the model's reasoning ability over alternative task situations are also under exploited. In order to evaluate the planning ability of both multi-modal and counterfactual aspects, we propose ActPlan-1K. ActPlan-1K is a multi-modal planning benchmark constructed based on ChatGPT and household activity simulator iGibson2. The benchmark consists of 153 activities and 1,187 instances. Each instance describing one activity has a natural language task description and multiple environment images from the simulator. The gold plan of each instance is action sequences over the objects in provided scenes. Both the correctness and commonsense satisfaction are evaluated on typical VLMs. It turns out that current VLMs are still struggling at generating human-level procedural plans for both normal activities and counterfactual activities. We further provide automatic evaluation metrics by finetuning over BLEURT model to facilitate future research on our benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
