arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.21764cs.LG
收藏

Q($γ_H$): 多折扣Q学习中的熵引导视界混合

Q($γ_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning

School of Computer Science, University of Leeds(利兹大学计算机学院)

Arwa Albelaihi, Abdulrahman Altahhan

AI总结:

本研究针对Q学习固定折扣因子无法适配不同训练需求的问题,提出多视界Q学习框架Q($γ_H$),通过熵引导的自适应混合权重实现视界切换,在延迟信用任务上较基线有显著改进,且兼容标准值学习方法。

摘要:

Q学习中固定的折扣因子会在整个训练过程中施加单一的规划视界,这与早期探索和长期信用分配的不同需求不符。我们提出Q($γ_H$),这是一种多视界Q学习框架,它维护多个与折扣相关的头,并在控制时使用源自熵的置信信号自适应地组合这些头。由此产生的依赖于状态的混合权重会随着策略置信度的提高,将控制从较短视界的头转移到较长视界的头,从而在不修改单个贝尔曼更新的情况下实现自适应视界混合。在经典控制和深度强化学习基准测试中,与强大的固定折扣基线相比,Q($γ_H$)在延迟信用任务上表现出最强的改进。通过将视界适应与底层学习更新解耦,Q($γ_H$)为强化学习中的自适应规划视界控制提供了一个轻量级框架,避免了手动选择折扣因子的需求,同时与标准值学习方法兼容。

英文摘要:

A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment. We propose Q($γ_H$), a multi-horizon Q-learning framework that maintains multiple discount-specific heads and adaptively combines them at control time using an entropy-derived confidence signal. The resulting state-dependent mixing weights shift control from shorter- to longer-horizon heads as policy confidence increases, enabling adaptive horizon mixing without modifying individual Bellman updates. Across classical control and deep reinforcement-learning benchmarks, Q($γ_H$) shows its strongest improvements on delayed-credit tasks relative to strong fixed-discount baselines. By decoupling horizon adaptation from the underlying learning updates, Q($γ_H$) provides a lightweight framework for adaptive planning-horizon control in reinforcement learning, avoiding the need for manual discount-factor selection while remaining compatible with standard value-learning methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑