arXiv 2510.056922025-10-07cs.ROcs.LG

面向视觉运动策略的Oracle引导掩码对比强化学习

Oracle-Guided Masked Contrastive Reinforcement Learning for Visuomotor Policies

School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航空航天工程学院）；College of Automation Engineering, Nanjing University of Aeronautics and Astronautics（南京航空航天大学自动化工程学院）

Yuhang Zhang, Jiaping Xiao, Chao Yan, Mir Feroskhan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出OMC-RL框架，通过两阶段解耦（掩码Transformer对比学习表示+oracle教师引导策略学习），提升视觉运动策略的样本效率、渐近性能与泛化能力。

摘要：

学习视觉运动策略的一种主流方法是采用强化学习，将高维视觉观测直接映射到动作指令。然而，高维视觉输入与敏捷机动输出的结合带来了长期存在的挑战，包括样本效率低和显著的仿真到现实差距。为解决这些问题，我们提出了Oracle-Guided Masked Contrastive Reinforcement Learning（OMC-RL），这是一个新颖的框架，旨在提高视觉运动策略学习的样本效率和渐近性能。OMC-RL明确地将学习过程解耦为两个阶段：上游表示学习阶段和下游策略学习阶段。在上游阶段，一个掩码Transformer模块通过时间建模和对比学习进行训练，以从序列视觉输入中提取具有时间感知和任务相关的表示。训练后，学习到的编码器被冻结，用于从连续帧中提取视觉表示，而Transformer模块则被丢弃。在下游阶段，一个具有全局状态信息特权访问权限的oracle教师策略在早期训练期间监督智能体，以提供信息丰富的指导并加速早期策略学习。随着训练的进行，这种指导逐渐减少，以允许独立探索。在模拟和真实环境中的大量实验表明，OMC-RL实现了优越的样本效率和渐近策略性能，同时提高了在多样且感知复杂的场景中的泛化能力。

英文摘要：

A prevailing approach for learning visuomotor policies is to employ reinforcement learning to map high-dimensional visual observations directly to action commands. However, the combination of high-dimensional visual inputs and agile maneuver outputs leads to long-standing challenges, including low sample efficiency and significant sim-to-real gaps. To address these issues, we propose Oracle-Guided Masked Contrastive Reinforcement Learning (OMC-RL), a novel framework designed to improve the sample efficiency and asymptotic performance of visuomotor policy learning. OMC-RL explicitly decouples the learning process into two stages: an upstream representation learning stage and a downstream policy learning stage. In the upstream stage, a masked Transformer module is trained with temporal modeling and contrastive learning to extract temporally-aware and task-relevant representations from sequential visual inputs. After training, the learned encoder is frozen and used to extract visual representations from consecutive frames, while the Transformer module is discarded. In the downstream stage, an oracle teacher policy with privileged access to global state information supervises the agent during early training to provide informative guidance and accelerate early policy learning. This guidance is gradually reduced to allow independent exploration as training progresses. Extensive experiments in simulated and real-world environments demonstrate that OMC-RL achieves superior sample efficiency and asymptotic policy performance, while also improving generalization across diverse and perceptually complex scenarios.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。