分块引导的Q学习
Chunk-Guided Q-Learning
AI总结:
本文提出Chunk-Guided Q-Learning,通过正则化细粒度单步批评者向基于分块的批评者学习,减少误差累积并保持细粒度价值传播,理论和实验均显示其优于单步和动作分块TD学习。
摘要:
在离线强化学习中,单步时间差分学习可能因长期时间差分误差积累而失效。动作分块的TD方法通过多步回溯缓解此问题,但会因限制策略类为开环动作序列而引入次优。为解决此权衡,我们提出Chunk-Guided Q-Learning (CGQ),一种单步TD算法,通过正则化细粒度单步批评者向基于时间扩展回溯训练的分块批评者学习。这在减少误差累积的同时保持细粒度价值传播。我们理论证明CGQ的批评者最优性界比单步或动作分块TD学习更紧。实验表明,CGQ在具有挑战性的长时间范围OGBench任务中表现优异,常优于单步和动作分块方法。
英文摘要:
In offline reinforcement learning (RL), single-step temporal-difference (TD) learning can suffer from bootstrapping error accumulation over long horizons. Action-chunked TD methods mitigate this by backing up over multiple steps, but can introduce suboptimality by restricting the policy class to open-loop action sequences. To resolve this trade-off, we present Chunk-Guided Q-Learning (CGQ), a single-step TD algorithm that guides a fine-grained single-step critic by regularizing it toward a chunk-based critic trained using temporally extended backups. This reduces compounding error while preserving fine-grained value propagation. We theoretically show that CGQ attains tighter critic optimality bounds than either single-step or action-chunked TD learning alone. Empirically, CGQ achieves strong performance on challenging long-horizon OGBench tasks, often outperforming both single-step and action-chunked methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。