arXiv 2605.221232026-05-21cs.RO

超越像素：从少量示范中学习不变的奖励以实现实世界机器人学

Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations

School of Computing and Data Science, The University of Hong Kong（计算科学与数据科学学院，香港大学）；LimX Dynamics Technology Co., Ltd（LimX动力技术有限公司）；Southern University of Science and Technology（南方科技大学）；Peking University（北京大学）；Zhejiang University（浙江大学）

Tengye Xu, Yangting Sun, Ziju Shen, Guanqi Chen, Zhen Fu, Chen yizhou, Hua Chen, Jia Pan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出了一种从少量示范中学习不变奖励的方法，以实现实世界机器人学中的泛化能力，通过发现行为不变量来改进奖励函数的设计，从而在多个任务中提升策略学习效果。

摘要：

设计能够超越受控实验室环境的奖励函数仍然是强化学习在机器人学中的基本挑战。在开放世界操纵问题中，单一任务可以通过不同的物体实例、位置和摄像头视角出现多种变体。最近基于视觉的奖励模型倾向于记忆特定的像素分布，并且无法超越其训练条件进行泛化。为了解决这个问题，我们提出了一种框架，该框架可以从最少的五个示范中学习不变的符号奖励函数。关键思想是将视觉特征拟合转向发现行为不变量：在多样化的视觉实例中保持不变的任务级属性。该框架有两个耦合的组件：一个结构化奖励公式，它编码任务级策略和物理约束，同时保持最优策略不变性；以及一个混合的符号-数值过程，该过程从示范中提炼这些不变量，而无需在线交互。在八个Meta-World任务和三个Franka操纵任务上的实验表明，我们的方法在过程对齐和策略展开排名能力方面优于基线方法，加速了下游策略学习。三个现实世界的出分布实验进一步表明，学习到的奖励能够零样本泛化到位置、视角和物体变体，使单一奖励表示能够在实践中重用于多种任务变体。

英文摘要：

Designing reward functions that generalize beyond controlled laboratory settings remains a fundamental challenge in reinforcement learning for robotics. In open-world manipulation problems, a single task can appear in numerous variants through different object instances, positions, and camera viewpoints. Recent vision-based reward models tend to memorize specific pixel distributions and fail to generalize beyond their training conditions. To address this, we propose a framework that learns invariant symbolic reward functions from as few as five demonstrations. The insight is to shift from visual feature-fitting to the discovery of behavioral invariants: task-level properties that remain constant across diverse visual instantiations. The framework has two coupled components: a structural reward formulation that encodes task-level strategies and physical constraints while preserving optimal policy invariance, and a hybrid symbolic-numerical procedure that distills these invariants from demonstrations without online interaction. Experiments on eight Meta-World tasks and three Franka manipulation tasks demonstrate that our method achieves stronger process alignment and policy rollout ranking abilities compared to baselines, accelerating downstream policy learning. Three real-world out-of-distribution experiments further show that the same learned reward generalizes zero-shot to position, viewpoint, and object variations, enabling a single reward representation to be reused across diverse task variants in practice.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。