从新颖性到模仿：用于离线强化学习的自蒸馏奖励
From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learning
AI总结：
ReLOAD通过自蒸馏生成内在奖励，解决离线强化学习中显式奖励标注的难题，实现稳健的策略学习并达到传统方法的性能水平。
摘要：
离线强化学习（RL）旨在从静态数据集中学习有效的策略，而无需进一步的智能体-环境交互。然而，其实际应用常常受到显式奖励标注需求的阻碍，这些标注可能成本高昂或难以回溯获得。为了解决这一问题，我们提出了ReLOAD（通过蒸馏进行离线奖励标注的强化学习），一种新的用于离线RL的奖励标注框架。与现有方法依赖复杂的对齐过程不同，我们的方法将随机网络蒸馏（RND）适应到使用专家演示生成内在奖励，通过简单的但有效的嵌入差异度量。首先，我们训练一个预测网络以模仿固定目标网络的嵌入，基于专家状态转移。随后，这些网络之间的预测误差为静态数据集中的每个转移提供奖励信号。这种机制在不需要手工奖励标注的情况下提供结构化的奖励信号。我们提供了一个正式的理论构造，为RND预测误差如何有效作为内在奖励提供见解，通过区分专家样式的转移。在D4RL基准测试中，ReLOAD实现了稳健的离线策略学习，并在传统奖励标注方法中取得了竞争性的性能。
英文摘要：
Offline Reinforcement Learning (RL) aims to learn effective policies from a static dataset without requiring further agent-environment interactions. However, its practical adoption is often hindered by the need for explicit reward annotations, which can be costly to engineer or difficult to obtain retrospectively. To address this, we propose ReLOAD (Reinforcement Learning with Offline Reward Annotation via Distillation), a novel reward annotation framework for offline RL. Unlike existing methods that depend on complex alignment procedures, our approach adapts Random Network Distillation (RND) to generate intrinsic rewards from expert demonstrations using a simple yet effective embedding discrepancy measure. First, we train a predictor network to mimic a fixed target network's embeddings based on expert state transitions. Later, the prediction error between these networks serves as a reward signal for each transition in the static dataset. This mechanism provides a structured reward signal without requiring handcrafted reward annotations. We provide a formal theoretical construct that offers insights into how RND prediction errors effectively serve as intrinsic rewards by distinguishing expert-like transitions. Experiments on the D4RL benchmark demonstrate that ReLOAD enables robust offline policy learning and achieves performance competitive with traditional reward-annotated methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
