从专家演示中进行奖励分配的最小要素
Minimal Ingredients for Reward Assignment from Expert Demonstrations
AI总结：
该研究探究从专家演示分配奖励的最小结构，通过32个基准测试和三种下游RL算法，发现离线场景仅需接近度，在线或多演示时需轻量级时间对应，倡导奖励设计的算法极简主义。
摘要：
从稀缺演示中进行奖励分配是离线和在线模仿学习中的关键挑战。一种常见且直观的策略是根据学习者轨迹与专家演示的匹配程度来分配奖励。尽管这一原理是许多现有方法的基础，但驱动性能的核心要素仍未得到系统探索。因此，我们提出问题：奖励分配必须编码的最小结构是什么，才能在各种设置中实现有效的下游强化学习（RL）性能？我们沿着两个设计轴解决这个问题：接近度近似和时间对齐。在涵盖离线和在线设置的32个基准测试中，以及使用三种下游RL算法，我们的实证结果表明：（1）在离线场景中，仅接近度就足以捕获有效离线RL所需的奖励结构；而（2）轻量级时间对应关系提供的增益在离线时较为温和，但在线或存在多个演示时至关重要。我们还补充了离线结果，提出了一种轻量级理论，用于描述简单接近度近似何时足够。总体而言，这些发现主张在引入复杂方案之前，在离线和在线模仿学习中采用奖励设计的算法极简主义。
英文摘要：
Reward assignment from scarce demonstrations is a key challenge in both offline and online imitation learning. A common and intuitive strategy assigns rewards according to how closely learner trajectories match expert demonstrations. Although this principle underlies many existing methods, the core ingredients that drive performance remain systematically underexplored. We therefore ask: what is the minimal structure that reward assignment must encode to achieve effective downstream RL performance across settings? We approach this question along two design axes: proximity approximation and temporal alignment. Across 32 benchmarks spanning offline and online settings, and with three downstream RL algorithms, our empirical findings suggest: (1) In offline regimes, proximity alone captures the reward structure necessary for effective offline RL, while (2) lightweight temporal correspondence provides consistent gains that are modest offline but essential online or in the presence of multiple demonstrations. We further complement our offline results with a lightweight theory characterizing when simple proximity approximation suffices. Overall, these findings advocate algorithmic minimalism in reward design before introducing complex schemes in both offline and online imitation learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
