PEAC：面向跨具身强化学习的无监督预训练
PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning
AI总结：
针对现有跨具身强化学习知识与任务耦合、难捕捉具身特征的问题，提出跨具身无监督强化学习框架CEURL及算法PEAC，通过内在奖励实现具身感知的任务无关预训练，显著提升跨具身泛化与适配性能。
摘要：
设计能够适配多种具身形态的通用智能体已成为强化学习（RL）领域的研究热点，这对将RL智能体部署到各类现实应用中至关重要。以往的跨具身RL方法聚焦于在特定任务中实现跨具身的知识迁移，这类方法得到的知识往往与对应任务高度耦合，无法充分捕捉不同具身形态的独特特征。为解决这一局限，我们提出跨具身无监督强化学习（CEURL）概念，借助无监督学习让智能体在无奖励环境中通过在线交互，获取具备具身感知、与任务无关的知识。我们将CEURL形式化为一种全新的受控具身马尔可夫决策过程（CE-MDP），并在CE-MDP框架下系统分析了CEURL的预训练目标。基于这些分析，我们开发了用于处理CEURL的新算法——预训练具身感知控制（PEAC），该算法融入了专为跨具身预训练设计的内在奖励函数。PEAC不仅为跨具身预训练提供了直观的优化策略，还能灵活与现有无监督RL方法结合，助力跨具身探索与技能发现。我们在模拟环境（如DMC、Robosuite）和现实环境（如足式运动）中开展了大量实验，结果表明PEAC显著提升了适配性能与跨具身泛化能力，证明其能有效应对CEURL的独特挑战。项目主页与代码见https://yingchengyang.github.io/ceurl。
英文摘要：
Designing generalizable agents capable of adapting to diverse embodiments has achieved significant attention in Reinforcement Learning (RL), which is critical for deploying RL agents in various real-world applications. Previous Cross-Embodiment RL approaches have focused on transferring knowledge across embodiments within specific tasks. These methods often result in knowledge tightly coupled with those tasks and fail to adequately capture the distinct characteristics of different embodiments. To address this limitation, we introduce the notion of Cross-Embodiment Unsupervised RL (CEURL), which leverages unsupervised learning to enable agents to acquire embodiment-aware and task-agnostic knowledge through online interactions within reward-free environments. We formulate CEURL as a novel Controlled Embodiment Markov Decision Process (CE-MDP) and systematically analyze CEURL's pre-training objectives under CE-MDP. Based on these analyses, we develop a novel algorithm Pre-trained Embodiment-Aware Control (PEAC) for handling CEURL, incorporating an intrinsic reward function specifically designed for cross-embodiment pre-training. PEAC not only provides an intuitive optimization strategy for cross-embodiment pre-training but also can integrate flexibly with existing unsupervised RL methods, facilitating cross-embodiment exploration and skill discovery. Extensive experiments in both simulated (e.g., DMC and Robosuite) and real-world environments (e.g., legged locomotion) demonstrate that PEAC significantly improves adaptation performance and cross-embodiment generalization, demonstrating its effectiveness in overcoming the unique challenges of CEURL. The project page and code are in https://yingchengyang.github.io/ceurl.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
