从语言学习不变语义表示实现可扩展策略泛化
Learning Invariable Semantical Representation from Language for Extensible Policy Generalization
摘要：
最近，将自然语言指令纳入强化学习（RL）以学习语义上有意义的表示并促进泛化引起了许多关注。然而，语言指令中的语义信息通常与特定于任务的状态信息纠缠在一起，这阻碍了语义不变和可重用表示的学习。在本文中，我们提出了一种学习这种表示的方法，称为元素随机化，该方法使用一组具有随机元素的环境从指令中提取任务相关但与环境无关的语义，例如，拓扑结构或纹理，但同样的语言指令。我们从理论上证明了通过随机化学习语义不变表示的可行性。在实践中，我们相应地开发了一个层次结构的政策，其中一个高层次的政策的目的是通过提出子目标作为语义不变的表示来调节的行为的目标条件下的低层次的政策。在具有挑战性的长时间任务上的实验表明：（1）我们的低级别策略可靠地概括了环境变化的任务;（2）我们的分层策略在看不见的新任务中表现出可扩展的概括性，这些新任务可以分解为几个可解决的子任务;（3）通过存储和重放语言轨迹作为简洁的策略表示，代理可以一次性完成任务，即，一旦一个成功的轨道已经实现。
英文摘要：
Recently, incorporating natural language instructions into reinforcement learning (RL) to learn semantically meaningful representations and foster generalization has caught many concerns. However, the semantical information in language instructions is usually entangled with task-specific state information, which hampers the learning of semantically invariant and reusable representations. In this paper, we propose a method to learn such representations called element randomization, which extracts task-relevant but environment-agnostic semantics from instructions using a set of environments with randomized elements, e.g., topological structures or textures, yet the same language instruction. We theoretically prove the feasibility of learning semantically invariant representations through randomization. In practice, we accordingly develop a hierarchy of policies, where a high-level policy is designed to modulate the behavior of a goal-conditioned low-level policy by proposing subgoals as semantically invariant representations. Experiments on challenging long-horizon tasks show that (1) our low-level policy reliably generalizes to tasks against environment changes; (2) our hierarchical policy exhibits extensible generalization in unseen new tasks that can be decomposed into several solvable sub-tasks; and (3) by storing and replaying language trajectories as succinct policy representations, the agent can complete tasks in a one-shot fashion, i.e., once one successful trajectory has been attained.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
