arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.01789cs.RO
收藏

RFS: 通过残差流引导的强化学习用于灵巧操作

RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulation

University of Washington, Paul G. Allen School of Computer Science & Engineering(华盛顿大学,保罗·G·艾伦计算机科学与工程学院);Amazon Frontier AI & Robotics(亚马逊前沿人工智能与机器人)

Entong Su, Tyler Westenbroek, Anusha Nagabandi, Abhishek Gupta

AI总结:

RFS通过残差流引导强化学习,实现高效适应预训练生成策略,提升灵巧操作任务的性能

摘要:

模仿学习已逐渐成为在机器人中实现序列决策制定的有效方法,即使在高维灵巧操作任务中也能取得强劲表现。最近的行为克隆方法进一步利用了表达能力强的生成模型,如扩散模型和流匹配,以表示多模态动作分布。然而,以这种方式预训练的策略往往表现出有限的泛化能力,并需要额外的微调才能在部署时实现稳健的性能。这种适应必须在保留预训练的全局探索优势的同时,能够快速纠正局部执行错误。我们提出了残差流引导(RFS),一种数据高效的强化学习框架,用于适应预训练的生成策略。RFS通过联合优化残差动作和潜在噪声分布来引导预训练的流匹配策略,使探索形式互补:通过残差修正实现局部细化,通过潜在空间调节实现全局探索。这种设计允许高效适应的同时保留预训练策略的表达结构。我们在灵巧操作任务中展示了RFS的有效性,显示在适应预训练基础策略时,在仿真和现实世界设置中均能实现高效的微调。项目网站:https://weirdlabuw.github.io/rfs

英文摘要:

Imitation learning has emerged as an effective approach for bootstrapping sequential decision-making in robotics, achieving strong performance even in high-dimensional dexterous manipulation tasks. Recent behavior cloning methods further leverage expressive generative models, such as diffusion models and flow matching, to represent multimodal action distributions. However, policies pretrained in this manner often exhibit limited generalization and require additional fine-tuning to achieve robust performance at deployment time. Such adaptation must preserve the global exploration benefits of pretraining while enabling rapid correction of local execution errors. We propose Residual Flow Steering(RFS), a data-efficient reinforcement learning framework for adapting pretrained generative policies. RFS steers a pretrained flow-matching policy by jointly optimizing a residual action and a latent noise distribution, enabling complementary forms of exploration: local refinement through residual corrections and global exploration through latent-space modulation. This design allows efficient adaptation while retaining the expressive structure of the pretrained policy. We demonstrate the effectiveness of RFS on dexterous manipulation tasks, showing efficient fine-tuning in both simulation and real-world settings when adapting pretrained base policies. Project website:https://weirdlabuw.github.io/rfs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑