用于符号自动机编码目标的无模型强化学习
Model-Free Reinforcement Learning for Symbolic Automata-encoded Objectives
AI总结：
针对强化学习依赖状态奖励易致策略不满足任务目标或不安全的问题，提出用符号自动机编码目标，定义非稀疏势能奖励提升收敛性，且能保证策略满足规范。
摘要：
强化学习（RL）是不确定环境下机器人路径规划的常用方法。然而，为RL智能体训练的控制策略严重依赖用户定义的基于状态的奖励函数。设计不佳的奖励可能导致策略获得最大奖励，但无法满足所需任务目标或不安全。已有多个使用时序逻辑和自动机等形式语言指定机器人高级任务规范（替代马尔可夫奖励）的例子。近期研究聚焦于从形式规范中推断基于状态的奖励；在此，目标是提供（概率）保证，即使用RL（通过推断奖励）学习的策略满足高级形式规范。这些技术的一个关键缺点是推断的奖励稀疏：智能体仅在任务完成时获得正奖励，否则无奖励，这自然导致RL期间收敛性差且方差高。本研究提出使用符号自动机编码目标的无模型强化学习：符号自动机是有界时序逻辑规范和自动机的泛化。此外，符号自动机的使用允许定义非稀疏的基于势能的奖励，通过经验塑造奖励表面，提升RL期间的收敛性。研究还表明，基于势能的奖励策略仍能获得最大化满足给定规范的策略。
英文摘要：
Reinforcement learning (RL) is a popular approach for robotic path planning in uncertain environments. However, the control policies trained for an RL agent crucially depend on user-defined, state-based reward functions. Poorly designed rewards can lead to policies that do get maximal rewards but fail to satisfy desired task objectives or are unsafe. There are several examples of the use of formal languages such as temporal logics and automata to specify high-level task specifications for robots (in lieu of Markovian rewards). Recent efforts have focused on inferring state-based rewards from formal specifications; here, the goal is to provide (probabilistic) guarantees that the policy learned using RL (with the inferred rewards) satisfies the high-level formal specification. A key drawback of several of these techniques is that the rewards that they infer are sparse: the agent receives positive rewards only upon completion of the task and no rewards otherwise. This naturally leads to poor convergence properties and high variance during RL. In this work, we propose using formal specifications in the form of symbolic automata: these serve as a generalization of both bounded-time temporal logic-based specifications as well as automata. Furthermore, our use of symbolic automata allows us to define non-sparse potential-based rewards which empirically shape the reward surface, leading to better convergence during RL. We also show that our potential-based rewarding strategy still allows us to obtain the policy that maximizes the satisfaction of the given specification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
