通过未来相关期权推广LTL指令
Generalizing LTL Instructions via Future Dependent Options
摘要：
在控制系统和机器人的许多实际应用中，线性时序逻辑（LTL）是一种广泛使用的任务规约语言，它具有一个组合语法，自然地诱导跨任务的时间扩展行为，包括条件和替代实现。LTL任务强化学习中的一个重要问题是学习任务条件策略，该策略可以zero-shot推广到训练中未观察到的新LTL指令。然而，由于符号观测通常是有损的，并且LTL任务可能具有较长的时间范围，因此以前的工作可能会遇到诸如训练采样效率低下以及所找到的解决方案的不可行性或次优性等问题。为了解决这些问题，本文提出了一种新的多任务强化学习算法，提高了学习效率和最优性。为了实现任务完成的全局最优性，我们建议通过一种新的离线策略方法来学习依赖于未来子目标的选项。为了更有效地传播满足未来子目标的回报，我们提出了训练一个多步价值函数的子目标序列的条件下，更新与蒙特卡洛估计的多步贴现收益。在三个不同领域的实验中，我们评估了LTL泛化能力的代理训练所提出的方法，显示其优势，比以前的代表性方法。
英文摘要：
In many real-world applications of control system and robotics, linear temporal logic (LTL) is a widely-used task specification language which has a compositional grammar that naturally induces temporally extended behaviours across tasks, including conditionals and alternative realizations. An important problem in RL with LTL tasks is to learn task-conditioned policies which can zero-shot generalize to new LTL instructions not observed in the training. However, because symbolic observation is often lossy and LTL tasks can have long time horizon, previous works can suffer from issues such as training sampling inefficiency and infeasibility or sub-optimality of the found solutions. In order to tackle these issues, this paper proposes a novel multi-task RL algorithm with improved learning efficiency and optimality. To achieve the global optimality of task completion, we propose to learn options dependent on the future subgoals via a novel off-policy approach. In order to propagate the rewards of satisfying future subgoals back more efficiently, we propose to train a multi-step value function conditioned on the subgoal sequence which is updated with Monte Carlo estimates of multi-step discounted returns. In experiments on three different domains, we evaluate the LTL generalization capability of the agent trained by the proposed method, showing its advantage over previous representative methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
