通过模拟过去来学习该做什么
Learning What To Do by Simulating the Past
摘要:
由于奖励函数很难指定,最近的工作集中在从人类反馈中学习策略。然而,这种方法受到获取这种反馈的费用的阻碍。最近的工作提出,智能体可以访问一个实际上免费的信息源:在人类活动的任何环境中,状态已经针对人类偏好进行了优化,因此智能体可以从状态中提取关于人类想要什么的信息。这种学习在原则上是可能的,但需要模拟可能导致观察状态的所有可能的过去轨迹。这在网格世界中是可行的,但我们如何将其扩展到复杂的任务?在这项工作中,我们表明,通过将学习到的特征编码器与学习到的逆模型相结合,我们可以使智能体能够及时地模拟人类的动作,以推断他们必须做什么。由此产生的算法是能够重现特定的技能,在MuJoCo环境给定的一个单一的状态采样从该技能的最佳策略。
英文摘要:
Since reward functions are hard to specify, recent work has focused on learning policies from human feedback. However, such approaches are impeded by the expense of acquiring such feedback. Recent work proposed that agents have access to a source of information that is effectively free: in any environment that humans have acted in, the state will already be optimized for human preferences, and thus an agent can extract information about what humans want from the state. Such learning is possible in principle, but requires simulating all possible past trajectories that could have led to the observed state. This is feasible in gridworlds, but how do we scale it to complex tasks? In this work, we show that by combining a learned feature encoder with learned inverse models, we can enable agents to simulate human actions backwards in time to infer what they must have done. The resulting algorithm is able to reproduce a specific skill in MuJoCo environments given a single state sampled from the optimal policy for that skill.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。