发表机构
University of Washington; NVIDIA（华盛顿大学; 英伟达）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对超大规模机器人控制RL的探索瓶颈，提出SGS自适应采样器，结合百万级并行仿真，解决多地形四足运动等难题，实现策略向真实硬件的零样本迁移。
AI 中文摘要
通用机器人需完成从敏捷运动到灵巧操作的各类任务。尽管仿真到现实的强化学习（RL）是实现该目标的有用工具，但当前RL流程依赖于大量工程化的、针对特定任务的结构先验，如塑形奖励和演示。近期研究表明，多样化的仿真器重置结合大规模并行仿真，可减轻部分操作任务的工程负担。然而，我们发现将该范式直接扩展到更精确或动态的任务仍非易事。仿真器重置虽有助于探索，但对该分布的均匀采样会使学习经验中越来越大的比例被浪费在策略已掌握或暂无法尝试的任务配置上，这使得难以看到并行环境扩展对RL的预期益处，因为批次中的大量学习信号在学习过程中被浪费。为缓解此问题，我们提出成功引导采样（Success Guided Sampling，SGS），一种简单的自适应采样器，将RL训练集中在策略能力前沿附近的任务配置上。这使大规模仿真RL能充分利用批次中的经验，实现向大规模并行仿真的更有效扩展。在使用多达2^20（超100万）个并行环境的实验中，SGS使RL解决了具有挑战性的多地形四足运动和接触丰富的装配任务，而这些任务是现有方法无法解决的。最后，我们将学习到的操作策略提炼为基于RGB的策略，并展示其在真实硬件上对多个具有挑战性的装配任务的零样本迁移。项目网站：this https URL。
英文摘要
General-purpose robots must perform a wide range of tasks from agile locomotion to dexterous manipulation. While sim-to-real reinforcement learning (RL) has proven to be a useful tool for this goal, current RL pipelines depend on engineering-heavy, per-task structural priors such as shaped rewards and demonstrations. Recent work has shown that diverse simulator resets, combined with massively parallel simulation, can alleviate much of this engineering burden on several manipulation problems. However, we find that naively scaling this paradigm to more precise or dynamic problems remains non-trivial. While simulator resets can help with exploration, uniformly sampling over this distribution wastes a growing fraction of learning experience on task configurations the policy has already mastered or cannot yet attempt. This makes it challenging to see the expected benefits of scaling parallel environments for RL, since much of the learning signal in a batch is wasted during learning. To mitigate this, we introduce Success Guided Sampling (SGS), a simple adaptive sampler that concentrates RL training on task configurations around the frontier of the policy's capabilities. Doing so allows large-scale simulated RL to make the most out of the experience in a batch, enabling much more effective scaling to large-scale parallel simulation. Across experiments using up to $2^{20}$ (over one million) parallel environments, SGS enables RL to solve challenging multi-terrain quadruped locomotion and contact-rich assembly tasks that prior methods fail to solve. Finally, we distill the learned manipulation policies into RGB-based policies and demonstrate zero-shot transfer to several challenging assembly tasks on real hardware. Project website: https://sgs-rl.github.io/.
CommentsCoRL 2026. Project website: https://sgs-rl.github.io/
