通用机器人策略的现实世界后训练面临两大挑战：部分可观测性使动作选择存在歧义，而离线后训练方法会延迟部署经验的反馈。我们提出Value as Memory（VaM，作为记忆的值）框架，通过值缓存和在线现实世界强化学习（RL）解决这些挑战。值缓存将动作条件化于近期值预测，作为任务进展记忆；在线更新通过结合离线演示、离策略干预和在线策略rollout，实现及时的策略改进，以实现样本高效的适应。因此，VaM从离线RL预热过渡到具有稀疏奖励的在线自主学习。在涉及可变形物体和接触丰富操作的长 horizon 任务中，VaM通过值缓存实现平均绝对成功率提升29.4%，通过在线自主学习实现比匹配离线训练高32.8%的提升。所评估的在线调度在相同梯度步骤预算下执行8倍多的策略迭代，且通过并发数据收集和训练，每次梯度步骤的记录时间降低2倍以上。
英文摘要：
Real-world post-training of generalist robot policies faces two challenges: partial observability makes action selection ambiguous, while offline post-training methods delay feedback from deployment experience. We present Value as Memory (VaM), a framework that addresses these challenges through value caching and online real-world reinforcement learning (RL). Value caching conditions actions on recent value predictions as task-progress memory, while online updates enable timely policy improvement by combining offline demonstrations, off-policy interventions, and on-policy rollouts for sample-efficient adaptation. As a result, VaM transitions from offline RL warmup to online autonomous learning with sparse rewards. Across long-horizon tasks involving deformable objects and contact-rich manipulation, VaM yields a mean absolute success-rate gain of $29.4\%$ with value caching, while achieving a $32.8\%$ gain with online autonomous learning over matched offline training. The evaluated online schedule performs $8\times$ as many policy iterations under the same gradient-step budget, with more than $2\times$ lower recorded time per gradient step with concurrent data collection and training.
Choose What to Manipulate: Revealing Data Scaling Laws in Bounding-Box Guided Policies for Semantic Manipulation
Center for Intelligent Control and Telescience, Tsinghua Shenzhen International Graduate School（智能控制与遥科学中心，清华大学深圳国际研究生院）；Beijing Xiaomi Robot Technology Co., Ltd（北京小米机器人科技有限公司）
LTLDiff: Finite Linear Temporal Logic-Guided Data Generation and Diffusion Policies for Multi-agent Robotic Manipulation
University of Toronto（多伦多大学）；A*STAR Centre for Frontier AI Research (A*STAR CFAR), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局前沿人工智能研究中心（A*STAR CFAR））
DemoSpeedup: Accelerating Visuomotor Policies via Entropy-Guided Demonstration Acceleration
Shanghai Qi Zhi Institute（上海启智研究院）；Tsinghua University IIIS（清华大学智能科学研究院）；Shanghai AI Lab（上海人工智能实验室）；University of Electronic Science and Technology of China（电子科技大学）
In this paper, we propose a method for training control policies for human-robot interactions such as handshakes or hand claps via Deep Reinforcement Learning. The policy controls a humanoid Shadow Dexterous Hand, attached to a robot arm. We propose a parameterizable multi-objective reward function that allows learning of a variety of interactions without changing the reward structure. The parameters of the reward function are estimated directly from motion capture data of human-human interactions in order to produce policies that are perceived as being natural and human-like by observers. We …
Recent advancement in combining trajectory optimization with function approximation (especially neural networks) shows promise in learning complex control policies for diverse tasks in robot systems. Despite their great flexibility, the large neural networks for parameterizing control policies impose significant challenges. The learned neural control policies are often overcomplex and non-smooth, which can easily cause unexpected or diverging robot motions. Therefore, they often yield poor generalization performance in practice. To address this issue, we propose adVErsarially Regularized pOlic…
BayesFP: Posterior Estimation for Flow-Based Policies via Feynman-Kac Sampling
School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；Australian Centre For Robotics, The University of Sydney（悉尼大学澳大利亚机器人中心）；College of Connected Computing, Vanderbilt University（范德堡大学互联计算学院）；NVIDIA（英伟达）
FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning
Machine Learning Research Lab, Volkswagen Group（大众集团机器学习研究实验室）；Technical University of Munich（慕尼黑技术大学）；Foundation Robotics Labs, Munich（慕尼黑基础机器人实验室）；Eötvös Loránd University（劳尔·艾尔多迪大学）
ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations
Thomas Lord Department of Computer Science, University of Southern California（南加州大学计算机科学系）；Amazon Robotics（亚马逊机器人技术）；Kim Jaechul School of Artificial Intelligence, KAIST（韩国科学技术院人工智能学院）
CoLF: Learning Consistent Leader-Follower Policies for Vision-Language-Guided Multi-Robot Cooperative Transport
Division of Information Science, Graduate School of Science and Technology, Nara Institute of Science and Technology (NAIST)（信息科学系，科学技术研究生学校，科学与技术国立研究所（NAIST））；École polytechnique fédérale de Lausanne (EPFL)（洛桑联邦理工学院（EPFL））
Guided Policy Search enables robots to learn control policies for complex manipulation tasks efficiently. Therein, the control policies are represented as high-dimensional neural networks which derive robot actions based on states. However, due to the small number of real-world trajectory samples in Guided Policy Search, the resulting neural networks are only robust in the neighbourhood of the trajectory distribution explored by real-world interactions. In this paper, we present a new policy representation called Generative Motor Reflexes, which is able to generate robust actions over a broade…
Model predictive control (MPC) is an effective method for controlling robotic systems, particularly autonomous aerial vehicles such as quadcopters. However, application of MPC can be computationally demanding, and typically requires estimating the state of the system, which can be challenging in complex, unstructured environments. Reinforcement learning can in principle forego the need for explicit state estimation and acquire a policy that directly maps sensor readings to actions, but is difficult to apply to unstable systems that are liable to fail catastrophically during training before an …
Towards Safe Imitation Learning via Potential Field-Guided Flow Matching
Department of Robotics, Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（机器人系，Mohamed bin Zayed大学人工智能学院）；Bosch Center for Artificial Intelligence（博世人工智能中心）；Division of Robotics, Perception, and Learning, KTH Royal Institute of Technology（机器人、感知与学习 division，皇家理工学院）
University of Southern California（南加州大学）；Google Brain（谷歌大脑）
We present a policy search method for learning complex feedback control policies that map from high-dimensional sensory inputs to motor torques, for manipulation tasks with discontinuous contact dynamics. We build on a prior technique called guided policy search (GPS), which iteratively optimizes a set of local policies for specific instances of a task, and uses these to train a complex, high-dimensional global policy that generalizes across task instances. We extend GPS in the following ways: (1) we propose the use of a model-free local optimizer based on path integral stochastic optimal cont…
Quadratic Program(QP) based state-feedback controllers, whose inequality constraints bound the rate of change of control barrier(CBFs) and lyapunov function with a class-$\mathcal{K}$ function of their values, are sensitive to the parameters of these class-$\mathcal{K}$ functions. The construction of valid CBFs, however, is not straightforward, and for arbitrarily chosen parameters of the QP, the system trajectories may enter states at which the QP either eventually becomes infeasible, or may not achieve desired performance. In this work, we pose the control synthesis problem as a differential…
Oracle-Guided Masked Contrastive Reinforcement Learning for Visuomotor Policies
School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航空航天工程学院）；College of Automation Engineering, Nanjing University of Aeronautics and Astronautics（南京航空航天大学自动化工程学院）
Imitation learning (IL) can generate computationally efficient sensorimotor policies from demonstrations provided by computationally expensive model-based sensing and control algorithms. However, commonly employed IL methods are often data-inefficient, requiring the collection of a large number of demonstrations and producing policies with limited robustness to uncertainties. In this work, we combine IL with an output feedback robust tube model predictive controller (RTMPC) to co-generate demonstrations and a data augmentation strategy to efficiently learn neural network-based sensorimotor pol…
In this paper, we continue our prior work on using imitation learning (IL) and model free reinforcement learning (RL) to learn driving policies for autonomous driving in urban scenarios, by introducing a model based RL method to drive the autonomous vehicle in the Carla urban driving simulator. Although IL and model free RL methods have been proved to be capable of solving lots of challenging tasks, including playing video games, robots, and, in our prior work, urban driving, the low sample efficiency of such methods greatly limits their applications on actual autonomous driving. In this work,…
High level declarative constraints provide a powerful (and popular) way to define and construct control policies; however, most synthesis algorithms do not support specifying the degree of randomness (unpredictability) of the resulting controller. In many contexts, e.g., patrolling, testing, behavior prediction,and planning on idealized models, predictable or biased controllers are undesirable. To address these concerns, we introduce the \emph{Entropic Reactive Control Improvisation} (ERCI) framework and algorithm which supports synthesizing control policies for stochastic games that are decla…
