EVLP：通过强化监督微调学习统一的具身视觉-语言规划器
EVLP:Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuning
AI总结：
本文提出统一具身视觉-语言规划器EVLP，通过动态感知预训练和强化监督微调联合建模语言推理与视觉生成，提升长时程操作任务中的多模态规划一致性与空间感知能力。
摘要：
在复杂的具身长时程操作任务中，有效的任务分解与执行需要将文本逻辑推理和视觉空间想象协同整合，以确保高效且准确的操作。现有方法未能采用统一的多模态规划生成框架，导致多模态规划不一致。为应对这一挑战，我们提出EVLP（具身视觉-语言规划器），这是一个创新的多模态统一生成框架，联合建模语言推理和视觉生成。我们的方法通过包含动态预训练和强化对齐的新型训练流程，实现长时程任务的多模态规划。我们的核心创新包括三个关键部分：1）统一多模态生成框架：在理解方面，我们将语义信息与空间特征集成，以提供全面的视觉感知；在生成方面，我们直接学习离散图像的联合分布以进行一步视觉合成，通过可学习的跨模态注意力机制实现协调的语言-视觉建模。2）动态感知预训练：我们提出一种双向动态对齐策略，采用逆动力学任务和前向动力学任务，在统一特征空间内有效增强多模态相关性。3）强化监督微调：在统一生成空间中进行基于指令的微调的同时，我们构建强化损失以对齐文本动作与生成图像之间的空间逻辑，使模型获得具有空间感知能力的多模态规划能力。
英文摘要：
In complex embodied long-horizon manipulation tasks, effective task decomposition and execution require synergistic integration of textual logical reasoning and visual-spatial imagination to ensure efficient and accurate operation. Current methods fail to adopt a unified generation framework for multimodal planning, lead to inconsistent in multimodal planning. To address this challenge, we present \textbf{EVLP (Embodied Vision-Language Planner)}, an innovative multimodal unified generation framework that jointly models linguistic reasoning and visual generation. Our approach achieves multimodal planning for long-horizon tasks through a novel training pipeline incorporating dynamic pretraining and reinforced alignment. Our core innovations consist of three key components: \textbf{1) Unified Multimodal Generation Framework}: For understanding, We integrate semantic information with spatial features to provide comprehensive visual perception. For generation, we directly learn the joint distribution of discrete images for one-step visual synthesis, enabling coordinated language-visual modeling through learnable cross-modal attention mechanisms. \textbf{2) Dynamic Perception Pretraining}: We propose a bidirectional dynamic alignment strategy employing inverse dynamics tasks and forward dynamics tasks, effectively strengthening multimodal correlations within a unified feature space. \textbf{3) Reinforced Supervised Fine-Tuning}: While conducting instruction-based fine-tuning in the unified generation space, we construct a reinforce loss to align the spatial logic between textual actions and generated images, enabling the model to acquire spatio-awared multimodal planning capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
