LLaPa：一个用于反事实感知程序化规划的视觉-语言模型框架
LLaPa: A Vision-Language Model Framework for Counterfactual-Aware Procedural Planning
AI总结：
本文提出视觉-语言模型框架LLaPa用于多模态程序化规划，结合任务-环境重排序器与反事实活动检索器增强推理，在ActPlan-1K和ALFRED基准上生成更优规划。
摘要：
尽管大型语言模型（LLMs）通过强大的推理能力推进了具身AI系统的程序化规划，但多模态输入与反事实推理的整合仍有待探索。为应对这些挑战，我们引入了LLaPa，一个专为多模态程序化规划设计的视觉-语言模型框架。LLaPa利用视觉-语言模型（VLMs）从文本任务描述和视觉环境图像中生成可执行的动作序列。此外，我们通过两个辅助模块增强了LLaPa以改善程序化规划。第一个模块是任务-环境重排序器（TER），它利用面向任务的分割来创建任务敏感的特征空间，将文本描述与视觉环境对齐，并突出程序化执行的关键区域。第二个模块是反事实活动检索器（CAR），它识别并强调潜在的反事实条件，从而增强模型在反事实场景中的推理能力。在ActPlan-1K和ALFRED基准上的大量实验表明，LLaPa生成了具有更优LCS和正确性的更高质量规划，超越了先进模型。代码和模型可在https://github.com/sunshibo1234/LLaPa获取。
英文摘要：
While large language models (LLMs) have advanced procedural planning for embodied AI systems through strong reasoning abilities, the integration of multimodal inputs and counterfactual reasoning remains underexplored. To tackle these challenges, we introduce LLaPa, a vision-language model framework designed for multimodal procedural planning. LLaPa generates executable action sequences from textual task descriptions and visual environmental images using vision-language models (VLMs). Furthermore, we enhance LLaPa with two auxiliary modules to improve procedural planning. The first module, the Task-Environment Reranker (TER), leverages task-oriented segmentation to create a task-sensitive feature space, aligning textual descriptions with visual environments and emphasizing critical regions for procedural execution. The second module, the Counterfactual Activities Retriever (CAR), identifies and emphasizes potential counterfactual conditions, enhancing the model's reasoning capability in counterfactual scenarios. Extensive experiments on ActPlan-1K and ALFRED benchmarks demonstrate that LLaPa generates higher-quality plans with superior LCS and correctness, outperforming advanced models. The code and models are available https://github.com/sunshibo1234/LLaPa.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
