DAPPER: 一种基于偏好判别性的政策到政策偏好强化学习，用于高效机器人技能获取
DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisition
AI总结：
DAPPER通过多策略轨迹比较和判别性学习，提升机器人技能获取的查询效率。
摘要：
基于偏好的强化学习（PbRL）通过简单的查询比较单一策略的轨迹来实现策略学习。虽然人类对这些查询的响应使得可以学习与人类偏好一致的策略，但PbRL存在查询效率低的问题，因为策略偏差限制了轨迹的多样性，并减少了可用于学习偏好的可区分查询数量。本文将偏好判别性定义为衡量人类判断哪条轨迹更接近其理想行为的难易程度的指标，作为提高查询效率的关键指标。为了解决这个问题，我们超越了单一策略内的比较，而是通过比较多个策略的轨迹生成查询，因为从头开始训练可以促进多样性而不受策略偏差影响。我们提出了Discriminability-Aware Policy-to-Policy Preference-Based Efficient Reinforcement Learning（DAPPER），该方法整合了偏好判别性和通过多个策略实现的轨迹多样化。DAPPER在每次奖励更新后从头开始训练新策略，并采用一个学习估计偏好判别性的判别器，使更可区分的查询得到优先采样。在训练过程中，它联合最大化偏好奖励和偏好判别性得分，鼓励发现高奖励且易于区分的策略。在模拟和现实世界腿部机器人环境中进行的实验表明，DAPPER在查询效率方面优于先前的方法，特别是在具有挑战性的偏好判别性条件下。一个补充视频有助于理解所提出框架及其实验结果，可在：https://youtu.be/lRwX8FNN8n4获取。
英文摘要：
Preference-based Reinforcement Learning (PbRL) enables policy learning through simple queries comparing trajectories from a single policy. While human responses to these queries make it possible to learn policies aligned with human preferences, PbRL suffers from low query efficiency, as policy bias limits trajectory diversity and reduces the number of discriminable queries available for learning preferences. This paper identifies preference discriminability, which quantifies how easily a human can judge which trajectory is closer to their ideal behavior, as a key metric for improving query efficiency. To address this, we move beyond comparisons within a single policy and instead generate queries by comparing trajectories from multiple policies, as training them from scratch promotes diversity without policy bias. We propose Discriminability-Aware Policy-to-Policy Preference-Based Efficient Reinforcement Learning (DAPPER), which integrates preference discriminability with trajectory diversification achieved by multiple policies. DAPPER trains new policies from scratch after each reward update and employs a discriminator that learns to estimate preference discriminability, enabling the prioritized sampling of more discriminable queries. During training, it jointly maximizes the preference reward and preference discriminability score, encouraging the discovery of highly rewarding and easily distinguishable policies. Experiments in simulated and real-world legged robot environments demonstrate that DAPPER outperforms previous methods in query efficiency, particularly under challenging preference discriminability conditions. A supplementary video that facilitates understanding of the proposed framework and its experimental results is available at: https://youtu.be/lRwX8FNN8n4
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
