arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2105.08666cs.LGcs.AI
收藏

通过稀疏性正则化的稀疏执行动作强化学习

Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularization

Nanjing University(南京大学);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院)

Jing-Cheng Pang, Tian Xu, Shengyi Jiang, Yu-Ren Liu, Yang Yu

AI总结:

本文针对强化学习中动作执行次数受限的问题,提出稀疏动作马尔可夫决策过程(SA-MDP)及动作稀疏性正则化(ASRE)算法,通过稀疏性评估与分布正则化优化策略,在稀疏执行动作任务及Atari游戏中优于基线。

摘要:

强化学习(RL)在具身控制、自动驾驶和金融交易等决策任务中展现了令人瞩目的性能。在许多决策任务中,智能体经常面临在有限预算下执行动作的问题。然而,经典RL方法通常忽视此类稀疏执行动作所带来的挑战。它们在问题表述和有效算法开发中均假设所有动作可以被无限次地执行。为解决RL中有限动作执行的问题,本文首先将问题形式化为稀疏动作马尔可夫决策过程(SA-MDP),其中动作空间中的特定动作只能被执行有限次数。随后,我们提出一种策略优化算法——动作稀疏性正则化(ASRE),该算法自适应地以不同偏好处理每个动作。ASRE通过两个步骤运作:首先,ASRE通过受限动作采样评估动作稀疏性。接着,ASRE通过动作分布正则化将稀疏性评估纳入策略学习。我们提供了理论识别,验证了ASRE收敛到正则化最优值函数。在具有已知稀疏执行动作的任务实验中,经典RL算法难以高效训练策略,而ASRE有效约束动作采样并优于基线方法。此外,我们表明ASRE能普遍提升Atari游戏的性能,展示了其广泛的适用性。

英文摘要:

Reinforcement learning (RL) has demonstrated impressive performance in decision-making tasks like embodied control, autonomous driving and financial trading. In many decision-making tasks, the agents often encounter the problem of executing actions under limited budgets. However, classic RL methods typically overlook the challenges posed by such sparse-executing actions. They operate under the assumption that all actions can be taken for a unlimited number of times, both in the formulation of the problem and in the development of effective algorithms. To tackle the issue of limited action execution in RL, this paper first formalizes the problem as a Sparse Action Markov Decision Process (SA-MDP), in which specific actions in the action space can only be executed for a limited time. Then, we propose a policy optimization algorithm, Action Sparsity REgularization (ASRE), which adaptively handles each action with a distinct preference. ASRE operates through two steps: First, ASRE evaluates action sparsity by constrained action sampling. Following this, ASRE incorporates the sparsity evaluation into policy learning by way of an action distribution regularization. We provide theoretical identification that validates the convergence of ASRE to a regularized optimal value function. Experiments on tasks with known sparse-executing actions, where classical RL algorithms struggle to train policy efficiently, ASRE effectively constrains the action sampling and outperforms baselines. Moreover, we present that ASRE can generally improve the performance in Atari games, demonstrating its broad applicability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑