HAPFI:基于融合信息的感知历史规划
HAPFI: History-Aware Planning based on Fused Information
AI总结:
针对具身指令跟随任务中忽略历史数据和跨模态信息的问题,提出HAPFI方法,通过互注意力融合历史RGB、边界框、子目标和指令等多模态信息,显著提升了智能体的动作规划与重规划能力。
摘要:
具身指令跟随(Embodied Instruction Following, EIF)是一项根据高级自然语言指令规划长序列子目标的任务,例如“冲洗一片生菜并放在叉子旁边的白色桌子上”。为了成功执行这些长周期任务,我们认为智能体在每步决策时必须考虑其过去,即历史数据。然而,近期EIF方法常忽略历史数据中的知识,且未有效利用跨模态信息。为此,我们提出基于融合信息的感知历史规划(HAPFI),有效利用智能体与环境交互时收集的多种模态历史数据。具体而言,HAPFI通过我们的互注意力融合(Mutually Attentive Fusion)方法有效融合多种模态,包括历史RGB观测、边界框、子目标和高级指令。通过多项对比实验,我们表明利用历史多模态信息的智能体在动作规划能力上超越了所有忽略历史数据的对比方法,从而能为下一步生成充分知情的动作计划。此外,我们提供了定性证据,强调利用历史多模态数据的重要性,特别是在智能体遭遇中间失败的场景中,展示了其强大的重规划能力。
英文摘要:
Embodied Instruction Following (EIF) is a task of planning a long sequence of sub-goals given high-level natural language instructions, such as "Rinse a slice of lettuce and place on the white table next to the fork". To successfully execute these long-term horizon tasks, we argue that an agent must consider its past, i.e., historical data, when making decisions in each step. Nevertheless, recent approaches in EIF often neglects the knowledge from historical data and also do not effectively utilize information across the modalities. To this end, we propose History-Aware Planning based on Fused Information (HAPFI), effectively leveraging the historical data from diverse modalities that agents collect while interacting with the environment. Specifically, HAPFI integrates multiple modalities, including historical RGB observations, bounding boxes, sub-goals, and high-level instructions, by effectively fusing modalities via our Mutually Attentive Fusion method. Through experiments with diverse comparisons, we show that an agent utilizing historical multi-modal information surpasses all the compared methods that neglect the historical data in terms of action planning capability, enabling the generation of well-informed action plans for the next step. Moreover, we provided qualitative evidence highlighting the significance of leveraging historical multi-modal data, particularly in scenarios where the agent encounters intermediate failures, showcasing its robust re-planning capabilities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。