面向具身智能体的环境理解视觉-语言模型
Environmental Understanding Vision-Language Model for Embodied Agent
AI总结：
本文提出EUEA框架，通过细调四项核心技能提升具身智能体的环境理解能力，结合恢复步骤和GRPO阶段提升任务执行效果，实验显示在ALFRED任务中性能显著优于行为克隆基线。
摘要：
视觉-语言模型（VLMs）在指令跟随的具身智能体中展现出强大的感知和推理能力。然而，尽管具备这些能力及泛化性能，它们在环境理解方面仍面临限制，常在交互中失败或依赖环境元数据。为此，我们提出名为环境理解具身智能体（EUEA）的新型框架，细调四项核心技能：1）物体感知以识别相关物体，2）任务规划以生成交互子目标，3）动作理解以判断成功可能性，4）目标识别以确定目标完成。通过细调VLMs与EUEA技能，我们的框架使指令跟随任务执行更加可靠。我们进一步引入恢复步骤，利用这些核心技能和一组相对策略优化（GRPO）阶段，以细化不一致的技能预测。恢复步骤采样替代动作以纠正失败案例，GRPO阶段细化不一致的技能预测。在ALFRED任务中，我们的VLM显著优于行为克隆基线，平均成功率提升8.86%。恢复和GRPO阶段提供额外3.03%的增益，进一步提升整体性能。最后，我们的技能层面分析揭示了封闭源和开源VLMs在环境理解方面的关键限制，并识别了有效智能体-环境交互所需的必要能力。
英文摘要：
Vision-language models (VLMs) have shown strong perception and reasoning abilities for instruction-following embodied agents. However, despite these abilities and their generalization performance, they still face limitations in environmental understanding, often failing on interactions or relying on environment metadata during execution. To address this challenge, we propose a novel framework named Environmental Understanding Embodied Agent (EUEA), which fine-tunes four core skills: 1) object perception for identifying relevant objects, 2) task planning for generating interaction subgoals, 3) action understanding for judging success likelihood, and 4) goal recognition for determining goal completion. By fine-tuning VLMs with EUEA skills, our framework enables more reliable task execution for instruction-following. We further introduce a recovery step that leverages these core skills and a group relative policy optimization (GRPO) stage that refines inconsistent skill predictions. The recovery step samples alternative actions to correct failure cases, and the GRPO stage refines inconsistent skill predictions. Across ALFRED tasks, our VLM significantly outperforms a behavior-cloning baseline, achieving an 8.86% improvement in average success rate. The recovery and GRPO stages provide an additional 3.03% gain, further enhancing overall performance. Finally, our skill-level analyses reveal key limitations in the environmental understanding of closed- and open-source VLMs and identify the capabilities necessary for effective agent-environment interaction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
