LACMA：基于元动作的语言对齐对比学习在具身指令遵循中的应用
LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following
AI总结：
本文提出LACMA方法，通过对比学习对齐智能体隐藏状态与指令，并引入元动作弥合语义鸿沟，在ALFRED未见环境中取得4.5%的成功率提升。
摘要：
当训练环境中包含测试环境时，端到端 Transformers 在具身指令遵循任务中展现出了令人印象深刻的成功率。然而，当部署于未见过的环境时，它们往往表现不佳。这种泛化能力的缺乏是由于智能体对自然语言指令中的细微变化不够敏感。为了缓解这一问题，我们提出通过对比学习将智能体的隐藏状态与指令进行显式对齐。尽管如此，高级语言指令与智能体低级动作空间之间的语义鸿沟仍然是一个障碍。因此，我们进一步引入了元动作这一新颖概念来弥合这一鸿沟。元动作是普遍存在的动作模式，可以从原始动作序列中解析出来。这些模式代表了更高级的语义，在直觉上与指令更为接近。当元动作被用作额外的训练信号时，智能体在未见环境中的泛化能力更好。与强大的多模态 Transformer 基线相比，我们在 ALFRED 具身指令遵循的未见环境中取得了 4.5% 的显著绝对成功率提升。进一步分析表明，对比目标与元动作在实现最佳结果上具有互补性，且所得智能体能更好地将其状态与相应指令对齐，使其更适合现实世界的具身智能体。代码可在以下地址获取：https://github.com/joeyy5588/LACMA。
英文摘要：
End-to-end Transformers have demonstrated an impressive success rate for Embodied Instruction Following when the environment has been seen in training. However, they tend to struggle when deployed in an unseen environment. This lack of generalizability is due to the agent's insensitivity to subtle changes in natural language instructions. To mitigate this issue, we propose explicitly aligning the agent's hidden states with the instructions via contrastive learning. Nevertheless, the semantic gap between high-level language instructions and the agent's low-level action space remains an obstacle. Therefore, we further introduce a novel concept of meta-actions to bridge the gap. Meta-actions are ubiquitous action patterns that can be parsed from the original action sequence. These patterns represent higher-level semantics that are intuitively aligned closer to the instructions. When meta-actions are applied as additional training signals, the agent generalizes better to unseen environments. Compared to a strong multi-modal Transformer baseline, we achieve a significant 4.5% absolute gain in success rate in unseen environments of ALFRED Embodied Instruction Following. Additional analysis shows that the contrastive objective and meta-actions are complementary in achieving the best results, and the resulting agent better aligns its states with corresponding instructions, making it more suitable for real-world embodied agents. The code is available at: https://github.com/joeyy5588/LACMA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
