ACE：具备因果感知熵正则化的离策略Actor-Critic算法
ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization
AI总结：
针对现有无模型RL忽略不同基础行为重要性差异的问题，提出带因果感知熵正则化的离策略ACE算法，结合休眠引导重置机制，在29项连续控制任务上性能显著优于基线，兼具有效性与高样本效率。
摘要：
现有无模型强化学习算法普遍忽略了策略学习过程中不同基础行为的重要性存在差异这一特性。基于这一洞察，我们探究了不同动作维度与奖励之间的因果关系，以此评估训练阶段各类基础行为的重要程度。我们提出了一种因果感知熵项，能够有效识别并优先探索具有高潜在影响的动作，从而实现高效探索。此外，为避免算法过度聚焦于特定基础行为，我们分析了梯度休眠现象，并引入了休眠引导的重置机制，以进一步提升方法的效能。我们提出的算法——ACE: Off-policy Actor-critic with Causality-aware Entropy regularization（具备因果感知熵正则化的离策略Actor-Critic算法），在覆盖7个领域的29项多样化连续控制任务中，相较无模型强化学习基线方法展现出显著的性能优势，凸显了该方法的有效性、通用性与高效样本利用率。基准测试结果与演示视频可访问https://ace-rl.github.io/查看。
英文摘要：
The varying significance of distinct primitive behaviors during the policy learning process has been overlooked by prior model-free RL algorithms. Leveraging this insight, we explore the causal relationship between different action dimensions and rewards to evaluate the significance of various primitive behaviors during training. We introduce a causality-aware entropy term that effectively identifies and prioritizes actions with high potential impacts for efficient exploration. Furthermore, to prevent excessive focus on specific primitive behaviors, we analyze the gradient dormancy phenomenon and introduce a dormancy-guided reset mechanism to further enhance the efficacy of our method. Our proposed algorithm, ACE: Off-policy Actor-critic with Causality-aware Entropy regularization, demonstrates a substantial performance advantage across 29 diverse continuous control tasks spanning 7 domains compared to model-free RL baselines, which underscores the effectiveness, versatility, and efficient sample efficiency of our approach. Benchmark results and videos are available at https://ace-rl.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
