通过麦克劳林展开稳定极值Q学习
Stabilizing Extreme Q-learning by Maclaurin Expansion
AI总结：
针对极值Q学习（XQL）因损失函数指数项导致的不稳定性及误差分布偏离Gumbel分布的问题，提出麦克劳林展开极值Q学习，通过展开损失函数提升稳定性并实现稳定性与最优性的权衡，在DM Control在线任务和D4RL离线任务中均取得了性能提升。
摘要：
在离线强化学习中，样本内学习方法已被广泛用于防止因评估数据集中分布外动作而导致的性能下降。极值Q学习（XQL）采用基于贝尔曼误差服从Gumbel分布假设的损失函数，使其能够以样本内方式对软最优值函数建模，在离线和在线强化学习场景中均展现出强大性能。然而，其损失函数中的指数项会引发不稳定性，且误差分布可能偏离Gumbel分布，这些问题仍待解决。因此，我们提出麦克劳林展开极值Q学习以增强稳定性。该方法将麦克劳林展开应用于XQL的损失函数，提升对大误差的稳定性；同时在行为策略下的值函数与软最优值函数间进行调整，根据展开阶数实现稳定性与最优性的权衡，并能将误差分布假设从正态分布调整为Gumbel分布。在DM Control的在线RL任务中，该方法显著提升了XQL的稳定性；在D4RL的多个离线RL任务中，其性能也得到了改善。
英文摘要：
In offline reinforcement learning, in-sample learning methods have been widely used to prevent performance degradation caused by evaluating out-of-distribution actions from the dataset. Extreme Q-learning (XQL) employs a loss function based on the assumption that Bellman error follows a Gumbel distribution, enabling it to model the soft optimal value function in an in-sample manner. It has demonstrated strong performance in both offline and online reinforcement learning settings. However, issues remain, such as the instability caused by the exponential term in the loss function and the risk of the error distribution deviating from the Gumbel distribution. Therefore, we propose Maclaurin Expanded Extreme Q-learning to enhance stability. In this method, applying Maclaurin expansion to the loss function in XQL enhances stability against large errors. This approach involves adjusting the modeled value function between the value function under the behavior policy and the soft optimal value function, thus achieving a trade-off between stability and optimality depending on the order of expansion. It also enables adjustment of the error distribution assumption from a normal distribution to a Gumbel distribution. Our method significantly stabilizes learning in online RL tasks from DM Control, where XQL was previously unstable. Additionally, it improves performance in several offline RL tasks from D4RL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
