选择前先量化：用于鲁棒强化学习的主动动力学偏好
Quantification before Selection: Active Dynamics Preference for Robust Reinforcement Learning
AI总结：
针对 Domain Randomization 训练的策略过于保守的问题，提出 Active Dynamics Preference (ADP) 方法，通过量化并主动选择高信息量、低密度的系统参数来稳定训练，显著提升了策略在四个机器人运动任务中对系统不一致性的鲁棒性。
摘要：
训练鲁棒的策略对于在真实世界系统中部署策略或处理不同动态系统间未知的动态失配至关重要。Domain Randomization (DR) 是一种简单且优雅的方法，它训练保守策略以应对不同的动态系统，而无需关于目标系统参数的专家知识。然而，现有研究表明，通过 DR 训练的策略往往过于保守，在目标域中表现不佳。我们的核心洞察是，具有不同参数的动态系统为策略提供了不同级别的难度，且在系统中表现良好的难度会随着策略的演化而不断变化。如果我们能够动态地主动为策略采样具有适当难度的系统，这将稳定训练过程并防止策略变得过于保守或过于乐观。为了实现这一想法，我们引入了 Active Dynamics Preference (ADP)，它量化了采样系统参数的信息量和密度。ADP 主动选择具有高信息量和低密度的系统参数。我们在四个机器人运动任务中验证了我们的方法，这些任务在训练和测试环境之间存在各种差异。大量结果表明，与几个基线相比，我们的方法对系统不一致性具有卓越的鲁棒性。
英文摘要：
Training a robust policy is critical for policy deployment in real-world systems or dealing with unknown dynamics mismatch in different dynamic systems. Domain Randomization~(DR) is a simple and elegant approach that trains a conservative policy to counter different dynamic systems without expert knowledge about the target system parameters. However, existing works reveal that the policy trained through DR tends to be over-conservative and performs poorly in target domains. Our key insight is that dynamic systems with different parameters provide different levels of difficulty for the policy, and the difficulty of behaving well in a system is constantly changing due to the evolution of the policy. If we can actively sample the systems with proper difficulty for the policy on the fly, it will stabilize the training process and prevent the policy from becoming over-conservative or over-optimistic. To operationalize this idea, we introduce Active Dynamics Preference~(ADP), which quantifies the informativeness and density of sampled system parameters. ADP actively selects system parameters with high informativeness and low density. We validate our approach in four robotic locomotion tasks with various discrepancies between the training and testing environments. Extensive results demonstrate that our approach has superior robustness for system inconsistency compared to several baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
