SCQ：使用Sigmoid有界熵稳定保守Q学习

中文摘要 AI 辅助

离线到在线强化学习降低了真实世界机器人学习的交互成本，但饱受持续的价值估计不稳定性困扰。现有方法通过悲观正则化、下界校准和架构归一化来解决这一问题，但一个被忽视的不稳定性来源在于熵公式：标准的对数熵项可能变为负值，从而破坏策略更新的稳定性。我们提出了SCQ（Sigmoid有界保守Q学习），该方法用严格保持正值的Sigmoid有界公式替换了该项。SCQ保留了保守Q正则化和基于回报的下界校准，在不牺牲探索的情况下稳定了策略优化。我们在D4RL（Minari）基准上，在单演示和标准数据集设置下评估了SCQ，并在仿真和真实世界视觉任务上进行了评估。SCQ在匹配或超越基线性能的同时，在基于状态和视觉的基准上展现出更稳定的训练动态，并成功迁移到四个真实机器人平台，包括操作、轮式、四足和人形系统。一项直接裁剪干预（去除负对数概率贡献）与梯度匹配的正分数控制表明，正性而非特定的分数形状本身是大部分改进的驱动因素。项目网站：此https URL。

英文摘要

Offline-to-online reinforcement learning reduces interaction cost for real-world robot learning but suffers from persistent value estimation instability. Existing methods address this through pessimistic regularization, lower-bound calibration, and architectural normalization, but an overlooked source of instability lies in the entropy formulation: the standard log-entropy term can become negative, destabilizing policy updates. We introduce SCQ (Sigmoid-Bounded Conservative Q-Learning), which replaces this term with a sigmoid-bounded formulation that stays strictly positive. SCQ retains conservative Q regularization and return-based lower-bound calibration, stabilizing policy optimization without sacrificing exploration. We evaluate SCQ on D4RL (Minari) benchmarks under both single-demonstration and standard dataset settings, as well as on simulation and real-world visual tasks. SCQ matches or exceeds baseline performance while exhibiting more stable training dynamics across state-based and visual benchmarks, and transfers to four real-robot platforms including manipulation, wheeled, quadruped, and humanoid systems. A direct clipping intervention that removes negative log-probability contributions, together with gradient-matched positive-score controls, indicates that positivity rather than a particular score shape alone drives much of the improvement. Project website: https://scq-rl.github.io.

发表机构

• Wuhan University（武汉大学）
• Anhui University（安徽大学）

机构由 AI 辅助整理，请以论文原文为准。