arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2110.07342cs.CLcs.LG
收藏

FILM:用模块化方法遵循语言中的指令

FILM: Following Instructions in Language with Modular Methods

Carnegie Mellon University(卡内基梅隆大学);Facebook AI Research(脸书人工智能研究院)

So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, Ruslan Salakhutdinov

摘要:

用于具体化指令遵循的最近方法通常使用模仿学习进行端到端训练。这通常需要使用专家轨迹和低级语言指令。这种方法假设神经状态将整合多模态语义来执行状态跟踪,建立空间记忆,探索和长期规划。相比之下,我们提出了一个模块化的方法与结构化的表示,(1)建立一个语义地图的场景和(2)执行探索与语义搜索策略,以实现自然语言的目标。我们的模块化方法实现了SOTA性能(24.46%),与以前的工作有很大的差距(8.17%绝对值),同时通过避开专家轨迹和低级指令使用更少的数据。然而,利用低级语言可以进一步提高我们的性能(26.49%)。我们的研究结果表明,一个明确的空间记忆和语义搜索策略可以提供一个更强大和更一般的表示状态跟踪和指导,即使在没有专家轨迹或低级别的指令。

英文摘要:

Recent methods for embodied instruction following are typically trained end-to-end using imitation learning. This often requires the use of expert trajectories and low-level language instructions. Such approaches assume that neural states will integrate multimodal semantics to perform state tracking, building spatial memory, exploration, and long-term planning. In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal. Our modular method achieves SOTA performance (24.46 %) with a substantial (8.17 % absolute) gap from previous work while using less data by eschewing both expert trajectories and low-level instructions. Leveraging low-level language, however, can further increase our performance (26.49 %). Our findings suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance, even in the absence of expert trajectories or low-level instructions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑