GPU并行多任务强化学习与演示引导的策略优化
A GPU-Parallel Framework for Heterogeneous Multi-Task Reinforcement Learning
AI总结：
提出一种将结构化操作任务族转化为GPU并行多任务强化学习基准的构建方法MT-LIBERO，并设计演示引导策略优化算法DGPO，结合重要性加权PPO与自适应行为克隆，实现异构任务套件的高效并行训练。
摘要：
大规模GPU并行强化学习已经改变了机器人仿真中可训练的内容，但大多数系统仍为每个任务优化一个专家策略。我们提出了一种构建方法，将结构化操作任务族转化为GPU并行多任务强化学习基准，并在Isaac Lab中使用LIBERO资产和任务谓词实例化为MT-Libero。该基准支持在异构任务套件上同时进行强化学习，具有并行渲染、物理随机化以及状态输入或视觉输入策略。为了使这种训练在稀疏成功信号和有限先验数据下变得实用，我们进一步提出了DGPO，一种在线演示引导方法，它将重要性加权PPO与对匹配演示动作的自适应行为克隆相结合。DGPO实现了对演示任务分布的可调偏好，在保持在线PPO的稳定性和在线改进优势的同时，优于无先验强化学习和现有的基于演示的方法。
英文摘要：
GPU-parallel simulation provides abundant robot interaction, but existing benchmarks rarely combine this scale with heterogeneous manipulation tasks and standardized multi-task RL evaluation. We introduce Hebero (Heterogeneous Benchmark for Robot Learning), a GPU-parallel Isaac Lab benchmark that enables efficient joint training and evaluation of a single policy across all 40 heterogeneous tasks. Scaling experiments show that increasing parallel replicas per task improves success under a fixed wall-clock budget. To support learning with sparse rewards and limited demonstrations, we propose Demonstration-Guided Policy Optimization (DGPO), which reuses demonstrations for dense tracking rewards and asymmetric value learning. Its shared stack supports controlled comparisons of learner-specific demonstration interfaces within PPO. Within DGPO framework, we introduce IW-ABC, which uses a lightweight per-task learning progress signal to coordinate adaptive behavior cloning (ABC), relaxing demonstration guidance with task progress, and importance weighting (IW), emphasizing lagging tasks in PPO updates. With 50 demonstrations per task, IW-ABC achieves 90.1% state-input mean success, outperforming the strongest baseline FAMO-ABC by 7.8 percentage points. Its visual counterpart reaches 93.5% mean success. Real-world experiments further demonstrate that a single multi-task policy trained in simulation can successfully perform four tasks on a physical Piper robot. The project page is available at https://hebero-rl.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
