PROF：一种基于大语言模型的离线模仿学习奖励代码偏好优化框架
PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning
AI总结：
针对现有离线模仿学习奖励估计过度简化奖励结构的问题，提出基于LLM的PROF框架，结合RPR策略自动优化奖励函数，在D4RL基准上性能优于或持平现有强基线。
摘要：
离线模仿学习（offline IL）可在无需显式奖励标注的情况下训练有效策略。近期方法尝试利用少量专家演示为无标注数据集估计奖励，但这类方法通常假设轨迹与专家演示的相似度和奖励正相关，过度简化了底层奖励结构。本文提出PROF框架，借助大语言模型（LLM）从自然语言描述和单条专家轨迹中生成并改进可执行的奖励函数代码。同时提出无需环境交互或强化学习训练的奖励函数质量评估与排序策略——奖励偏好排序（RPR），该策略计算奖励函数的优势分数，分数越高代表与专家偏好的对齐度越好。PROF通过交替执行RPR和基于文本的梯度优化，全自动完成下游策略学习最优奖励函数的选择与优化。在D4RL基准上的实验结果表明，PROF在多个数据集和领域上的表现超过或持平近期强基线方法，验证了该方法的有效性。
英文摘要：
Offline imitation learning (offline IL) enables training effective policies without requiring explicit reward annotations. Recent approaches attempt to estimate rewards for unlabeled datasets using a small set of expert demonstrations. However, these methods often assume that the similarity between a trajectory and an expert demonstration is positively correlated with the reward, which oversimplifies the underlying reward structure. We propose PROF, a novel framework that leverages large language models (LLMs) to generate and improve executable reward function codes from natural language descriptions and a single expert trajectory. We propose Reward Preference Ranking (RPR), a novel reward function quality assessment and ranking strategy without requiring environment interactions or RL training. RPR calculates the dominance scores of the reward functions, where higher scores indicate better alignment with expert preferences. By alternating between RPR and text-based gradient optimization, PROF fully automates the selection and refinement of optimal reward functions for downstream policy learning. Empirical results on D4RL demonstrate that PROF surpasses or matches recent strong baselines across numerous datasets and domains, highlighting the effectiveness of our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
