SMORE：用于离线目标条件强化学习的分数模型
SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learning
AI总结：
本文提出无判别器的离线GCRL方法SMORe，通过混合分布匹配与凸对偶目标学习动作分数，以有效利用次优数据，并在机器人操作和运动基准上显著优于现有方法。
摘要：
离线目标条件强化学习（GCRL）的任务是仅利用离线数据集并通过稀疏奖励函数，学习在环境中实现多个目标。离线GCRL对于开发能够利用既有数据集学习多样且可复用技能、而无需手工设计奖励函数的通用智能体至关重要。然而，当前基于监督学习和对比学习的GCRL方法在离线设定下往往并非最优。另一种GCRL视角针对占据匹配进行优化，但需要学习一个判别器，随后将其作为下游RL的伪奖励。所学判别器中的误差可能级联放大，对所得策略产生负面影响。我们从混合分布匹配这一新视角提出了一种新颖的GCRL方法，由此得到无需判别器的方法：SMORe。其关键洞见是将GCRL的占据匹配视角与凸对偶形式相结合，推导出一个能够更好利用次优离线数据的学习目标。SMORe学习分数或非归一化密度，用以表示在某状态下采取某动作对于达成特定目标的重要性。SMORe具有原则性；我们在由机器人操作和运动任务（包括高维观测）组成的完全离线GCRL基准上进行了大量实验，结果表明SMORe能够以显著优势超越最先进基线。
英文摘要：
Offline Goal-Conditioned Reinforcement Learning (GCRL) is tasked with learning to achieve multiple goals in an environment purely from offline datasets using sparse reward functions. Offline GCRL is pivotal for developing generalist agents capable of leveraging pre-existing datasets to learn diverse and reusable skills without hand-engineering reward functions. However, contemporary approaches to GCRL based on supervised learning and contrastive learning are often suboptimal in the offline setting. An alternative perspective on GCRL optimizes for occupancy matching, but necessitates learning a discriminator, which subsequently serves as a pseudo-reward for downstream RL. Inaccuracies in the learned discriminator can cascade, negatively influencing the resulting policy. We present a novel approach to GCRL under a new lens of mixture-distribution matching, leading to our discriminator-free method: SMORe. The key insight is combining the occupancy matching perspective of GCRL with a convex dual formulation to derive a learning objective that can better leverage suboptimal offline data. SMORe learns scores or unnormalized densities representing the importance of taking an action at a state for reaching a particular goal. SMORe is principled and our extensive experiments on the fully offline GCRL benchmark composed of robot manipulation and locomotion tasks, including high-dimensional observations, show that SMORe can outperform state-of-the-art baselines by a significant margin.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
