Can-Do！用于基于大型多模态模型的具身规划的数据集与神经符号接地框架
Can-Do! A Dataset and Neuro-Symbolic Grounded Framework for Embodied Planning with Large Multimodal Models
AI总结：
本文提出 Can-Do 基准数据集评估具身规划，并构建 NeuroGround 神经符号框架，通过结合环境状态感知与符号规划引擎，有效解决了 GPT-4V 等模型在视觉感知与推理上的瓶颈。
摘要：
大型多模态模型在视觉和语言任务中展现出了令人印象深刻的解决问题能力，并具备编码广泛世界知识的潜力。然而，对于这些模型而言，在真实环境中进行感知、推理、规划和行动仍然是一个开放性挑战。在这项工作中，我们引入了 Can-Do，这是一个基准数据集，旨在通过比以往数据集更多样化、更复杂的场景来评估具身规划能力。我们的数据集包含 400 个多模态样本，每个样本由自然语言用户指令、描绘环境的视觉图像、状态变化以及相应的行动计划组成。该数据涵盖了常识知识、物理理解和安全意识等多个方面。我们的细粒度分析表明，包括 GPT-4V 在内的最先进模型在视觉感知、理解和推理能力方面面临瓶颈。为了解决这些挑战，我们提出了 NeuroGround，这是一个神经符号框架，它首先将计划生成接地于感知到的环境状态中，然后利用符号规划引擎来增强模型生成的计划。实验结果证明了我们的框架与强基线相比的有效性。我们的代码和数据集可在 https://embodied-planning.github.io 获取。
英文摘要：
Large multimodal models have demonstrated impressive problem-solving abilities in vision and language tasks, and have the potential to encode extensive world knowledge. However, it remains an open challenge for these models to perceive, reason, plan, and act in realistic environments. In this work, we introduce Can-Do, a benchmark dataset designed to evaluate embodied planning abilities through more diverse and complex scenarios than previous datasets. Our dataset includes 400 multimodal samples, each consisting of natural language user instructions, visual images depicting the environment, state changes, and corresponding action plans. The data encompasses diverse aspects of commonsense knowledge, physical understanding, and safety awareness. Our fine-grained analysis reveals that state-of-the-art models, including GPT-4V, face bottlenecks in visual perception, comprehension, and reasoning abilities. To address these challenges, we propose NeuroGround, a neurosymbolic framework that first grounds the plan generation in the perceived environment states and then leverages symbolic planning engines to augment the model-generated plans. Experimental results demonstrate the effectiveness of our framework compared to strong baselines. Our code and dataset are available at https://embodied-planning.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
