arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.11110cs.ROcs.AI
收藏

ResWM:基于视觉强化学习的残差动作世界模型

ResWM: Residual-Action World Model for Visual RL

University of California, San Diego(加州大学圣地亚哥分校);Texas A&M University-Commerce(德克萨斯A&M大学-科摩斯)

Jseen Zhang, Gabriel Adineera, Jinzhou Tan, Jinoh Kim

AI总结:

ResWM通过将控制变量从绝对动作转换为残差动作,提升了视觉强化学习中世界模型的预测能力和稳定性,实现了更高效的样本利用和更平滑的控制策略。

摘要:

从原始视觉观测中学习预测性世界模型是强化学习(RL)中的核心挑战,特别是在机器人和连续控制领域。传统基于模型的RL框架直接将未来预测条件化于绝对动作,这使得优化不稳定:最优动作分布是任务相关的,事先未知,往往导致振荡或低效的控制。为此,我们引入了残差动作世界模型(ResWM),一种新的框架,将控制变量从绝对动作转换为残差动作——相对于前一步的增量调整。这种设计与现实世界控制的内在平滑性一致,减少了有效搜索空间,并稳定了长周期规划。为进一步增强表示,我们提出了一种观察差分编码器,明确建模相邻帧之间的变化,产生紧凑的潜在动态,这些动态自然与残差动作耦合。ResWM被集成到一种类似于Dreamer的潜在动态模型中,仅需最小的修改和没有额外的超参数。想象回放和策略优化都在残差动作空间中进行,使探索更平滑,控制方差更低,规划更可靠。在DeepMind控制套件上的实验证明,ResWM在样本效率、渐近回报和控制平滑性方面实现了持续改进,显著超越了Dreamer和TD-MPC等强大基线。除了性能外,ResWM产生的动作轨迹更加稳定和节能,这一特性对部署在现实世界中的机器人系统至关重要。这些发现表明,残差动作建模提供了一种简单但强大的原则,将RL算法的进步与机器人学的实际需求联系起来。

英文摘要:

Learning predictive world models from raw visual observations is a central challenge in reinforcement learning (RL), especially for robotics and continuous control. Conventional model-based RL frameworks directly condition future predictions on absolute actions, which makes optimization unstable: the optimal action distributions are task-dependent, unknown a priori, and often lead to oscillatory or inefficient control. To address this, we introduce the Residual-Action World Model (ResWM), a new framework that reformulates the control variable from absolute actions to residual actions -- incremental adjustments relative to the previous step. This design aligns with the inherent smoothness of real-world control, reduces the effective search space, and stabilizes long-horizon planning. To further strengthen the representation, we propose an Observation Difference Encoder that explicitly models the changes between adjacent frames, yielding compact latent dynamics that are naturally coupled with residual actions. ResWM is integrated into a Dreamer-style latent dynamics model with minimal modifications and no extra hyperparameters. Both imagination rollouts and policy optimization are conducted in the residual-action space, enabling smoother exploration, lower control variance, and more reliable planning. Empirical results on the DeepMind Control Suite demonstrate that ResWM achieves consistent improvements in sample efficiency, asymptotic returns, and control smoothness, significantly surpassing strong baselines such as Dreamer and TD-MPC. Beyond performance, ResWM produces more stable and energy-efficient action trajectories, a property critical for robotic systems deployed in real-world environments. These findings suggest that residual action modeling provides a simple yet powerful principle for bridging algorithmic advances in RL with the practical requirements of robotics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑